Microsoft Israel R&D unveils AI tools for better meeting summaries and a smart health agent

At Microsoft Israel R&D’s AI Playground conference, developments were presented that will allow you to better understand summaries of meetings you missed, ask an AI agent questions about your health, and analyze video in real time.

Microsoft Israel R&D held its annual conference – AI Playground – to present developments in AI, health, and security. As part of the conference, the company’s various teams showcased the developments they are working on at the local development center.

Microsoft’s health team presented the Healthcare Agent Service. This is a custom AI agent for a health fund (or other health organization), which allows a doctor or caregiver to converse with it in natural language for explanations of visit summaries, appointment scheduling, or task automation. The information is based on the data of that health organization. Today, Microsoft is already working in Israel with various health funds to implement similar services.

Another development saves doctors from having to type during patient appointments. This platform, called Dragon Copilot, is also AI-based. By securely analyzing and transcribing the conversation between doctor and patient, it provides accurate clinical documentation and reporting. It can also automatically retrieve medical information and generate referral letters or prescriptions. Estimates indicate that the system saves 50% to 60% of the time currently spent on clinical documentation, allowing medical professionals to focus on their patients rather than typing on a computer.

Our daily lives are full of virtual meetings that take up a lot of time and are not always conducted effectively. According to data presented in Microsoft’s Global Work Report, most employees (58%) say they have difficulty brainstorming in online meetings or catching up if they join a meeting late (57%). Additionally, 55% said that understanding the meeting’s conclusions and next steps is unclear in the meeting summary, and 56% report difficulty summarizing what happened in the meeting. Since February 2020, people are attending three times as many virtual meetings and calls per week (192%) compared to before.

Microsoft’s development for the Teams app, based on its AI assistant – Copilot – can analyze visual content presented during the meeting. The development allows it to read and refer to (in comments and summaries) what is shared visually on the screen, whether it is a presentation, website, or document. So, if a participant missed a slide in a shared presentation, they can ask Copilot to fill in the gap. Copilot will summarize the content shared on the screen, highlight critical points from the conversation, and generate entirely new content based on the meeting (for example, it can create a table from data presented in a presentation).

Additionally, the team demonstrated how the Copilot Teams recognizes what is written on a whiteboard, whether it is handwriting, a drawn graph, or a table. This is relevant not only for office or work meetings but also for using Teams in classroom lessons.

Some of the information analysis capabilities in security cameras are performed in the cloud, but in many industries there is a preference or regulatory requirement to keep critical information within the organization and not send it to the cloud. A tool called Azure Video Indexer, enabled by Arc, can analyze video content streamed from the client’s security cameras to provide personalized insights in real time.