DEV Community

humbertojaimes
humbertojaimes

Posted on • Edited on

Mobile App backend using Azure Functions and Face API

#serverlessseptember
Humberto Jaimes - humberto@humbertojaimes.net

This article is part of #ServerlessSeptember. You'll find other helpful articles, detailed tutorials, and videos in this all-things-Serverless content collection. New articles are published every day — that's right, every day — from community members and cloud advocates in the month of September.

Find out more about how Microsoft Azure enables your Serverless functions at https://docs.microsoft.com/azure/azure-functions/.

The Cognitive Locator project, known publicly as 'Busca.me', is a project dedicated to reporting and finding missing persons. The project was founded due to the disasters related to the earthquake of September 19, 2017, which affected multiple states in Mexico. At first, the project was solely focused on finding or reporting people who went missing as a result of the earthquake. However, now that the project has grown, it not only aims to support the people affected at that time, but to anyone who is going through this devastating situation.

The source code of the Cognitive Locator project is under the MIT license in https://github.com/humbertojaimes/cognitive-locator

The project backend was created using Azure Functions. And this is the project architecture

Alt Text

This post will explain how to create two functions. Both functions will have similar functionality as the Cognitive Locator backend core functions.

  • An HTTP function that register a device in an Azure Notification Hub.

  • A Blob Storage function that analyze a photo and report the result to a device using a Push Notification.

Alt Text

Note: At this time the post is only intended to explain the Azure Functions part. In the future I can create the Azure Notification Hub configuration and the Xamarin posts.

The original project has the Xamarin Sample

Prerequisites:

  • An Azure Account
  • A provisioned a configured Azure Notification Hub instance
  • A provisioned an Azure Face API service

Device Registration Function

Creating a new project

For this tutorial I am going to use Visual Studio 2019 For Mac.

  1. Create a new Project

Alt Text

1.1 Select "Azure Functions" in the "Cloud" tab

Alt Text

  1. Write "DeviceInstallationRegistration" as the function name and choose the "HttpTrigger"

Alt Text

  1. For this demo, select the "Anonymous" access level.

Alt Text

  1. To finalize the creation of the project use "ServelessSeptember" as the project and solution name.

Alt Text

  1. In the new project create a "Settings" class with the following content. 
public class Settings
    {
        public static string AzureWebJobsStorage = Environment.GetEnvironmentVariable("AzureWebJobsStorage");

        public static string FaceAPIKey = Environment.GetEnvironmentVariable("Vision_API_Subscription_Key");
        public static string Zone = Environment.GetEnvironmentVariable("Vision_API_Zone");


        public static string NotificationAccessSignature = Environment.GetEnvironmentVariable("NotificationHub_Access_Signature");
        public static string NotificationHubName = Environment.GetEnvironmentVariable("NotificationHub_Name");


    }
Enter fullscreen mode Exit fullscreen mode

This class is intended to get the keys and connections strings from the Azure Functions portal configuration.

Programming the function

  1. Before writing code, we need to install the "Microsoft.Azure.NotificationHubs" NuGet package.

Alt Text

  1. We need a class that contains the device information.

Create a folder "Domain" and a class "DeviceInformation" class inside it.

public class DeviceInstallation
    {
        public string InstallationId { get; set; }

        public string Platform { get; set; }

        public string PushChannel { get; set; }

    }
Enter fullscreen mode Exit fullscreen mode
  1. In a "Helpers" folder create a class "NotificationsHelper".

This is the content of the class

 public static class NotificationsHelper
    {

        //Notification Hub settings
        public static string ConnectionString = Settings.NotificationAccessSignature;
        public static string NotificationHubPath = Settings.NotificationHubName;

        // Initialize the Notification Hub
        static NotificationHubClient hub = NotificationHubClient.CreateClientFromConnectionString(ConnectionString, NotificationHubPath);

        /// <summary>
        /// Receive the information to register a new mobile device in the Azure Notification Hub
        /// </summary>
        /// <param name="deviceUpdate">The device information</param>
        /// <param name="log"></param>
        /// <returns></returns>
        public static async Task RegisterDevice(DeviceInstallation deviceUpdate, ILogger log)
        {
            Installation installation = new Installation();
            installation.InstallationId = deviceUpdate.InstallationId;
            installation.PushChannel = deviceUpdate.PushChannel;
            switch (deviceUpdate.Platform)
            {
                case "apns":
                    installation.Platform = NotificationPlatform.Apns;
                    break;
                case "fcm":
                    installation.Platform = NotificationPlatform.Fcm;
                    break;
                default:
                    throw new Exception("Invalid Channel");
            }
            installation.Tags = new List<string>();
            await hub.CreateOrUpdateInstallationAsync(installation);
            log.LogInformation("Device was registered");
        }


        /// <summary>
        /// Register a device to receive specific notifications related to a report
        /// </summary>
        /// <param name="installationId">Identifier of the device</param>
        /// <param name="requestId">The report identifier</param>
        /// <param name="log"></param>
        /// <returns></returns>
        public static async Task AddToRequest(string installationId, string requestId, ILogger log)
        {
            await AddTag(installationId, $"requestId:{requestId}", log);
            log.LogInformation($"Device was registered in the request {requestId}");
        }


        /// <summary>
        /// Remove a device from the notificartions related to a report
        /// </summary>
        /// <param name="installationId">Identifier of the device</param>
        /// <param name="requestId">The report identifier</param>
        /// <param name="log"></param>
        /// <returns></returns>
        public static async Task RemoveFromRequest(string installationId, string requestId, ILogger log)
        {
            await RemoveTag(installationId, $"requestId:{requestId}", log);
            log.LogInformation($"Device was removed from the request {requestId}");
        }



        /// <summary>
        /// Remove a device from an specific tag in the Azure Notification Hub
        /// </summary>
        /// <param name="installationId">The device identifier</param>
        /// <param name="tag">The Azure Notification Hub Tag</param>
        /// <param name="log"></param>
        /// <returns></returns>
        private static async Task RemoveTag(string installationId, string tag, ILogger log)
        {
            try
            {
                Installation installation = await hub.GetInstallationAsync(installationId);
                if (installation.Tags == null)
                {
                    if (installation.Tags.Contains(tag))
                        installation.Tags.Remove(tag);
                    await hub.CreateOrUpdateInstallationAsync(installation);
                }
            }
            catch (Exception ex)
            {
                log.LogInformation(ex.Message);
            }
        }


        /// <summary>
        /// Add a device to an specific tag in the Azure Notification Hub
        /// </summary>
        /// <param name="installationId">The device identifier</param>
        /// <param name="tag">The Azure Notification Hub Tag</param>
        /// <param name="log"></param>
        /// <returns></returns>
        private static async Task AddTag(string installationId, string newTag, ILogger log)
        {
            try
            {
                Installation installation = await hub.GetInstallationAsync(installationId);
                if (installation.Tags == null)
                    installation.Tags = new List<string>();
                installation.Tags.Add(newTag);
                await hub.CreateOrUpdateInstallationAsync(installation);
            }
            catch (Exception ex)
            {
                log.LogInformation(ex.Message);
            }
        }


        /// <summary>
        /// Remove a device from the Azure Notification Hub
        /// </summary>
        /// <param name="installationId">The device identifier</param>
        /// <returns></returns>
        public static async Task RemoveDevice(string installationId)
        {
            await hub.DeleteInstallationAsync(installationId);
        }


        /// <summary>
        /// Send a push notification about an specific report id. At this moment is only supported 1 device per report
        /// </summary>
        /// <param name="text">The notification message that will be displayed by the mobile device</param>
        /// <param name="requestId">the report identifier</param>
        /// <param name="installationId">The device identifier</param>
        /// <param name="log"></param>
        /// <returns></returns>
        public static async Task SendNotification(string text, string requestId, string installationId, ILogger log)
        {
            try
            {
                Installation installation = await hub.GetInstallationAsync(installationId);

                if (installation.Platform == NotificationPlatform.Fcm)
                {
                    var json = string.Format("{{\"data\":{{\"message\":\"{0}\"}}}}", text);
                    await hub.SendFcmNativeNotificationAsync(json, $"requestid:{requestId}");
                    log.LogInformation($"FCM notification was sent");
                }
                else
                {
                    var json = string.Format("{{\"aps\":{{\"alert\":\" {0}\"}}}}", text);
                    await hub.SendAppleNativeNotificationAsync(json, $"requestid:{requestId}");
                    log.LogInformation($"Apple notification was sent");
                }
            }
            catch (Exception ex)
            {
                log.LogInformation(ex.Message);
            }
        }


        /// <summary>
        /// Send a notification to all Apple and Firebase devices in the notification hub
        /// </summary>
        /// <param name="text">The notification message that will be displayed by the mobile device </param>
        /// <param name="log"></param>
        /// <returns></returns>
        public static async Task SendBroadcastNotification(string text, ILogger log)
        {
            try
            {
                var json = string.Format("{{\"data\":{{\"message\":\"{0}\"}}}}", text);
                await hub.SendFcmNativeNotificationAsync(json);
                log.LogInformation($"FCM notification was sent");

            }
            catch (Exception ex) //If there aren't FCM devices registered in the hub, it throws an error
            {
                log.LogInformation(ex.Message);
            }


            try
            {
                var json = string.Format("{{\"aps\":{{\"alert\":\" {0}\"}}}}", text);
                await hub.SendAppleNativeNotificationAsync(json);
                log.LogInformation($"Apple notification was sent");
            }
            catch (Exception ex) //If there aren't Apple devices registered in the hub, it throws an error
            {
                log.LogInformation(ex.Message);
            }
        }
    }
Enter fullscreen mode Exit fullscreen mode
  1. And finally the code of the function. 
[FunctionName("DeviceNotificationsRegistration")]
        public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "put", Route = "devicenotificationsregistrations/")]HttpRequestMessage req, ILogger log)
        {
            try
            {
                log.LogInformation("New device registration incoming");
                var content = await req.Content.ReadAsStringAsync();
                DeviceInstallation deviceUpdate = await req.Content.ReadAsAsync<DeviceInstallation>();
                await NotificationsHelper.RegisterDevice(deviceUpdate, log);
                log.LogInformation("New device registered");
                return req.CreateResponse(HttpStatusCode.OK);
            }
            catch (Exception ex)
            {
                log.LogInformation($"Error during device registration: {ex.Message}");
            }
            return req.CreateErrorResponse(HttpStatusCode.InternalServerError, "Error during device registration");
        }
Enter fullscreen mode Exit fullscreen mode

Person Registration Function

  1. Create a "FaceClientHelper" in the "Helpers" folder.

In the original project, the helper contains several methods that interact with the Face API. For demo purposes, this sample only has one method.

https://github.com/humbertojaimes/cognitive-locator/blob/master/source/CognitiveLocator.Functions/Client/FaceClient.cs

public class FaceClientHelper
    {
        private string FaceAPIKey = Settings.FaceAPIKey;
        private string Zone = Settings.Zone;

        /// <summary>
        /// Analyze a photo using the Face API
        /// </summary>
        /// <param name="url">The image url</param>
        /// <returns></returns>
        public async Task<List<JObject>> DetectFaces(String url)
        {
            using (var client = new HttpClient())
            {
                var service = $"https://{Zone}.api.cognitive.microsoft.com/face/v1.0/detect";
                client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", FaceAPIKey);
                byte[] byteData = Encoding.UTF8.GetBytes("{'url':'" + url + "'}");
                using (var content = new ByteArrayContent(byteData))
                {
                    content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
                    var httpResponse = await client.PostAsync(service, content);

                    if (httpResponse.StatusCode == HttpStatusCode.OK)
                    {
                        List<JObject> result = JsonConvert.DeserializeObject<List<JObject>>(await httpResponse.Content.ReadAsStringAsync());
                        return result;
                    }
                }
            }
            return null;
        }


    }
Enter fullscreen mode Exit fullscreen mode
  1. Create a new function in the project

Alt Text

  1. Choose the blob trigger and write "PersonRegistration" as the function name.

Set the connection string of the blob, for this sample we are use the same storage that comes with the function, so the connection string is "AzureWebJobsStorage"

Alt Text

  1. The code of the function is the following 
private static FaceClientHelper client_face = new FaceClientHelper();

        [FunctionName("PersonRegistration")]
        public static async Task Run([BlobTrigger("images/{name}.{extension}")]CloudBlockBlob blobImage, string name, string extension, ILogger log)
        {

            log.LogInformation($"Image: {name}.{extension}");
            string notificationMessage = "Error";
            string json = string.Empty;
            var deviceId = blobImage.Metadata["deviceid"];


            try
            {

                await NotificationsHelper.AddToRequest(deviceId, name, log);
                log.LogInformation($"uri {blobImage.Uri.AbsoluteUri}");
                log.LogInformation($"Zone {Settings.Zone}");

                //determine if image has a face
                List<JObject> list = await client_face.DetectFaces(blobImage.Uri.AbsoluteUri);

                //validate image extension 
                if (blobImage.Properties.ContentType != "image/jpeg")
                {
                    log.LogInformation($"no valid content type for: {name}.{extension}");
                    await blobImage.DeleteAsync();

                    notificationMessage = "Incorrect Image Format";
                    await NotificationsHelper.SendNotification(notificationMessage, name, deviceId, log);
                    return;
                }

                //if image has no faces
                if (list.Count == 0)
                {
                    log.LogInformation($"there are no faces in the image: {name}.{extension}");
                    await blobImage.DeleteAsync();
                    notificationMessage = "The are not faces in the photo";
                    await NotificationsHelper.SendNotification(notificationMessage, name, deviceId, log);
                    return;
                }

                //if image has more than one face
                if (list.Count > 1)
                {
                    log.LogInformation($"multiple faces detected in the image: {name}.{extension}");
                    await blobImage.DeleteAsync();
                    notificationMessage = "Multiple faces detected in the image";
                    await NotificationsHelper.SendNotification(notificationMessage, name, deviceId, log);
                    return;
                }

            }
            catch (Exception ex)
            {
                // await blobImage.DeleteAsync();

                log.LogInformation($"Error in file: {name}.{extension} - {ex.Message}");
                notificationMessage = "Error in file registration";
                await NotificationsHelper.SendNotification(notificationMessage, name, deviceId, log);
                return;
            }

            log.LogInformation("person registered successfully");
            notificationMessage = "Person registered successfully";
            await NotificationsHelper.SendNotification(notificationMessage, name, deviceId, log);
        }
    }
Enter fullscreen mode Exit fullscreen mode

Publishing the function

  1. Using the Visual Studio For Mac Wizzard we can publish our function.

Alt Text

Alt Text

Alt Text

  1. Using the Azure Storage Explorer (https://azure.microsoft.com/es-mx/features/storage-explorer/) create a new blob container for the "images".

Alt Text

Alt Text

For demo purposes, we are going to set the access level to public.

Alt Text

Alt Text

Alt Text

Configuring the function settings

  1. Using the azure portal, in the published function "Platform Features" option select "configuration"

Alt Text

Add the following settings in the application settings section.

  • "Vision_API_Subscription_Key" - The Face API access key
  • "Vision_API_Zone" - The Face API Azure Region
  • "NotificationHub_Access_Signature" - The Notification Hub connection string
  • "NotificationHub_Name" - The Notification Hub name

Alt Text

Testing the Functions

  1. We can register a new device using PostMan (https://www.getpostman.com/).

Get the function URL from the Azure portal

Alt Text

Alt Text
The following json contains the structure expected by the function

{
    "InstallationId": "87653f8dc8e80kiuy",
    "Platform": "fcm",
    "PushChannel": "Cafg:APA91bFPcN01WnfXqRMQhsSySWVY6fgIggW3lpZ_E1tPL94pdChZb0-dIxdhyh9WxNyoxIWggDPctYyOTqRKpssqG5mjr-7nOJxKwBdDPJrmz8b-xt-B5Xna66S4IZRJpAcqNea6biv_"
}
Enter fullscreen mode Exit fullscreen mode

In PostMan send a Put request using the URL and the json body

Alt Text

If everything works as expected, we can see this result in the function console that we can see in the Azure Portal.

Alt Text

  1. For the blob function test, the easiest way is using the Azure Storage Explorer.

Upload a new photo of a person face.

Alt Text

The first upload will cause an error in the function because the lack of the metadata.

Alt Text

Add the metadata to the blob using in the blob properties. Using the same device id that you use in the postman json.

Alt Text

Alt Text

Copy the blob and paste it. Then overwrite the original blob

Alt Text

Alt Text

Alt Text

Because this copy contains the metadata, the function will show a success message.

Alt Text

Top comments (0)

Read next

mikeyoung44 profile image

AI and the Black Box Problem: How Machine Learning Challenges Mathematical Proof Verification

Mike Young -

mikeyoung44 profile image

LLM Test Generators Miss Critical Bugs Due to Design Flaws, Study Shows

Mike Young -

taxicongo1 profile image

Understanding the Core of Data Privacy

Oh Waugh -

mikeyoung44 profile image

New AI Pattern-Matching Method Simplifies and Improves Data Generation

Mike Young -