Will the development of new AI applications within the Microsoft ecosystem become easier in the future? That’s a good question, because Azure AI Studio is currently in preview. The documentation and marketing videos are promising a lot.

But what’s behind the scenes? That’s why I’m writing this blog post to share my first impressions with you.

Benefits of Azure AI Studio

Ok, let me start and answer an important question: Why should I use Azure AI Studio?

Microsoft docs says I should use AI Studio to build generative AI applications on an enterprise-grade platform. Furthermore, I can use the AI Studio to interact with my projects in a code-first approach via Azure AI SDK and Azure AI CLI. Furthermore, I can seamlessly explore, build, test, and deploy using cutting-edge AI tools and ML models.

In addition, AI Studio provides enterprise-grade security, is a collaborative environment for teams to work together, and facilitates scalability for transforming proof of concepts into full-fledged production. Finally, the studio helps me to continuously monitor and improve my AI applications.

Sounds good, but I think I should form my own opinion…

Welcome in AI Studio

I start and open the URL https://ai.azure.com/ in my browser. Afterwards, this screen appears:

The home screen shows me some options where I can start building new AI applications. Interesting, when I’m scrolling down in the page, I discover the Microsoft’s model catalog:

It seems to me that Microsoft highlights here the available popular Large Language Models. I move over to Explore and see a lot more information:

Ah, I can search here for LLMs that fulfil my requirements. Furthermore, I can compare the existing models here:

Oh, there is also a Prompt catalog in AI studio:

Wow, I can use this prompt as an inspiration for my own AI applications. Even better, I can jump directly from the prompt examples into AI studios playground:

But before I can do this, I must create my first project…

Setup my first Project in AI Studio

Now, Azure AI Studio asks me to create a new project. I’m entering as Project name for this first impression MMe2k. In addition, I must select Create a new resource. This is because Azure AI Studio doesn’t find an existing component in my Azure subscription.

Afterwards, I define my resource name, resource group, select my Azure subscription, and the location of my resources. My favorite location for AI in Europe is currently Sweden Central. This is because most of the Azure OpenAI models are available in this region. This includes popular models like dall-e-3, gpt-4, and gpt-4-32k.

In addition, I can select my existing AI service deployments such as my Azure OpenAI and my Azure AI Search instances:

I continue and see, there is a Storage Account to store my training information, an Azure Key Vault to store credentials in a secure way, and an Application Insights together with a Log Analytics Workspace to monitor my AI applications:

After clicking on Create a project, I have to wait until everything is deployed.

Project Overview

Now my first project is ready in AI studio. My project overview currently provides basic information. No resource, just the description and the navigation menu:

Ok, the project description is obviously not so impressive. More interesting to me are Tools and Components which I find on the left side of my navigation bar.

Playground

First, I’m navigation into the Playground, where I can tryout the generative AI like in Open AI studio. The Chat interface allows me to test chat conversations. Furthermore, I can include my own data as RAG (retrieval augmented generation). In addition, I can adjust here also my LLM parameters:

Next, I have a look into Images. As expected, I can use here Dall-e to generate images based on my given prompt:

Finally, I’ll switch to Completions. This playground allows me to set up variables for my text completions:

Here I discover also, I can customize configured playground in a prompt flow:

After clicking on Customize in prompt flow, AI studio ask me about a name for my flow:

Prompt Flow

…and my configured playground example is opened here:

This is amazing. Behind the scenes, the AI studio picked up all the information and built me an AI application. Furthermore, I can use the prompt flow studio to finetune and debug my AI application. In addition, I can use later my created prompt flow during evaluation. Wow, I also see a visual presentation of my AI application as graph.

In summary, AI Studio has turned my playground example into a reusable service and gives me tools to further develop my AI application professionally. Finally, I can deploy here my developed application.

Evaluation

Also very interesting is the Evaluation aspect of AI studio. Here I can measure my created and configured AI applications. For this I have two options. I can choose between Metric evaluations and Manual evaluations:

In Manual evaluations I can define or use datasets of own question and answer pairs. This is to prepare my evaluation data. Afterwards, my AI is executed with configured parameters, the given system prompt, and my question as input. The result is displayed as output, and I can evaluate manually with thumb up or down:

For Metric evaluations I can choose also to set up question answer pairs based on my data:

Next, I can select here my previously created prompt flow:

Furthermore, I can select and configure predefined metrics. How cool is that!

Like the manual evaluation, I can also upload my evaluation datasets:

Then, I’m mapping my dataset information and the flow output to the metrics:

After a while, my evaluations are executed automatically:

… and can be compared:

As you see, unfortunately my configured evaluation failed. I guess I must investigate later into this problem…

Deployments

Another interesting part of AI Studio is the list of my deployments. You see from my screenshot the well-known LLMs like gpt-35-turbo and gpt-4 which are related to my configured Azure Open AI resource.

In addition, you see also at the bottom my project deployments:

Yes, this deployment was previously created from my prompt flow:

This is cool, because I automatically get an endpoint for my AI application that I can use from my applications. I’ll give it a try from my VS Code:

…and the answer is:

Very impressive!

Summary

Overall, I must say: Azure AI Studio seems to be a one-stop shop for creating new AI applications. What I’ve seen makes perfect sense to me. My first impression is that I really like the combination of playground, prompt flow and evaluation.

First, the playground allows me to prototype and test a new AI application. I can choose between text completions, chat, or image generation. Secondly, I can directly create from my playground a prompt flow which is my AI application. Furthermore, this allows me to optimize my AI application in a more professional way. Finally, I can setup manual and metric driven evaluations to measure my changes.

Also, the operational aspect is covered. This is because I can manage my project-related AI deployments within AI studio. In other words, a good combination of useful tools that belongs together.