Telegram bot and use of Google Cloud Vision

Hello! Recently, I already wrote an article about integrating my bot with IBM Watson, and in this article I’ll review integration with Google Cloud Vision to recognize cats and describe in more detail the insides of my bot.

A little background:

My bot worked quite successfully for a couple of months using IBM Watson recognition, but then a google cloud vision article appeared on Habré and it turned out that Google recognized images better than IBM. On the same day, I logged into the Google cloud platform developer console and started rewriting the cat moderation unit in my bot.

With a bit of searching, I found a suitable C # example on the GoogleCloudPlatform github . I changed the authentication from the example and made it from the json file with the private key, which I took in the “service accounts” section of the console.

authorization code on c #

private VisionService service;
        privatestring _JsonPath = @"C:\BOTS\fcatsbot\json.json";       
        private VisionService CreateAuthorizedClient(string JsonPath)
        {
            GoogleCredential credential =
                GoogleCredential.FromStream(new FileStream(JsonPath, FileMode.Open));            
            // Inject the Cloud Vision scopesif (credential.IsCreateScopedRequired)
            {
                credential = credential.CreateScoped(new[]
                {
                    VisionService.Scope.CloudPlatform
                });
            }
            var res = new VisionService(new BaseClientService.Initializer
            {
                HttpClientInitializer = credential,
                GZipEnabled = false
            });
            return res;
        }

Next, I redid the image moderation (label detection). In the example on the githaba, DetectLabels works with a file, and I needed to work with a link that I received from Telegram servers in order not to store image files in myself. I save only file_id in the database, which gives a good increase in the speed of work.

label detection code

privateasync Task<IList<AnnotateImageResponse>> DetectLabels(
            VisionService vision, string imageUrl)
        {
            // Convert image to Base64 encoded for JSON ASCII text based request   
            MemoryStream ms = new MemoryStream();
            using (var client = new HttpClient())
            {
                Stream imageBytes = await client.GetStreamAsync(imageUrl);
                imageBytes.CopyTo(ms);
            }
            byte[] imageArray = ms.ToArray();
            string imageContent = Convert.ToBase64String(imageArray);
            // Post label detection request to the Vision API// [START construct_request]var responses = vision.Images.Annotate(
                new BatchAnnotateImagesRequest()
                {
                    Requests = new[] {
                    new AnnotateImageRequest() {
                        Features = new []
                        { new Feature()
                            { Type =
                          "LABEL_DETECTION"}
                            },
                        Image = new Image() { Content = imageContent }
                    }
               }
                }).Execute();
            ms.Dispose();
            return responses.Responses;
        }

Then I look for in Responses if there is a label with a description of the cat, with a rating of more than 0.6, and thus, I determine if there is a cat in the image transferred to the bot:

work with labels

foreach (var response in responses.Responses)
{
  foreach (var label in response.LabelAnnotations)
  {
    double _score = label.Score == null ? 0 : Convert.ToDouble(label.Score.Value);
    varclass = label.Description.Trim();
    if (class .Contains("kitten") || class .Contains("cat") ) && (_score > 0.60))
    {
      HasCatOrKittenClass = true;//moderation OK
    }
  }
}

Here is the code to work with the Telegram API to get a link to an image from file_id, I used the library on a C # telegram bot :

telegram getFile

var file = await MainParams.TGBot.GetFileAsync(fileid);
var file_path = file.FilePath;
var urlImage = "https://api.telegram.org/file/bot" + MainParams.bot_token + "/" + file_path;

And when I send an image to a user using sendPhoto, I simply pass the saved file_id to the second parameter.

Thus, it turns out that when a user sends his photo of a cat for moderation (or uses thecatapi.com for this), I save only file_id in the database and later use it to get a link to the picture on Telegram servers and to send to users using sendPhoto. And image recognition using Google cloud vision works more accurately than IBM Watson

Tags: