October 4, 2018 at 14:53How we developed the AR application for reviewing historical places
Recently, we combined old technologies with modern technologies, what came of it read under the cut.
Augmented Reality
Augmented reality apps as city guides are a well-known theme and implemented by many developers. This direction of using AR was one of the first, as it allows you to use all the obvious possibilities of augmented reality: show users information about buildings, give information about the work of the institution and acquaint with the sights. At the last hackathon, which was held inside the company, several projects using augmented reality were presented, and we came up with the idea to create an AR-application that will show what a landmark or historical place looked like in the past. To do this, combine modern augmented reality technologies with old photographs. For example, facing the St. Isaac’s Cathedral,
The mechanics of work are as follows: the application displays the specified historical places and attractions of the city on the map, displays brief information about them, with the help of notifications notifies the user that he is not far from an interesting point. When a person approaches a historical monument at a distance of 40 meters, the AR mode becomes available. At the same time, the camera opens, and brief information about the objects is displayed directly in the space surrounding the user. The latter has the ability to interact with virtual objects: by touching the card of a historical place, you can proceed to view the album with images.
It would seem that the application is very simple, but even here there were some pitfalls. I will not bore you with a story about the implementation of trivial things like downloading data from a server or displaying points on a map, I will go straight to the functions that caused the problems.
Problem 1. Floating Points
So, the first thing to do was to place marker points in space in accordance with the actual location of historical places relative to the current location and the direction of the user's gaze.
To get started, we decided to use the already-prepared library for iOS: ARKit-CoreLocation . The project lies on GitHub in the public domain, contains, in addition to the code of the main classes, integration examples and allows us to complete the task of interest in a couple of hours. It is only necessary to feed the library the coordinates of the points and the image used as a marker.
Not surprisingly, this ease had to be paid. Marker points constantly floated in space: either they climbed to the ceiling, or they were drawn somewhere underfoot. Not every user would agree to catch the AR-object in focus for several minutes to get acquainted with the information that interests him.
As it turned out, many faced this library bug, but a solution has not yet been found. The code on GitHub, unfortunately, has not been updated for more than six months, so I had to bypass it.
We tried using the altitude instead of the fixed altitude in coordinates, which the LocationManager returned for the user's current position. However, this did not completely eliminate the problem. Data coming from Location Manager began to jump with a spread of up to 60 meters, as soon as the device was twisted in hand. As a result, the picture was unstable, which, of course, did not suit us again.
As a result, it was decided to abandon the ARKit-CoreLocation library and place points in space on their own. The article ARKit and CoreLocation, written by Christopher Web-Orenstein, helped a lot in this. I had to spend a little more time and refresh some mathematical aspects in my memory, but the result was worth it: AR-objects were finally in their places. After that, it remains only to scatter them along the Y axis so that the labels and points are easier to read, and put a correspondence between the distance from the current position to the point and the Z coordinate of the AR object, so that information about the nearest historical places is in the foreground.
It was necessary to calculate the new SCNNode position in space, focusing on the coordinates:
let place = PlaceNode()
let locationTransform = MatrixHelper.transformMatrix(for: matrix_identity_float4x4, originLocation: curUserLocation, location: nodeLocation, yPosition: pin.yPos, shouldScaleByDistance: false)
let nodeAnchor = ARAnchor(transform: locationTransform)
scene.session.add(anchor: nodeAnchor)
scene.scene.rootNode.addChildNode(place)
The following functions were added to the MatrixHelper class:
class MatrixHelper {
static func transformMatrix(for matrix: simd_float4x4, originLocation: CLLocation, location: CLLocation, yPosition: Float) -> simd_float4x4 {
let distanceToPoint = Float(location.distance(from: originLocation))
let distanceToNode = (10 + distanceToPoint/1000.0)
let bearing = GLKMathDegreesToRadians(Float(originLocation.coordinate.direction(to: location.coordinate)))
let position = vector_float4(0.0, yPosition, -distanceToNode, 0.0)
let translationMatrix = MatrixHelper.translationMatrix(with: matrix_identity_float4x4, for: position)
let rotationMatrix = MatrixHelper.rotateAroundY(with: matrix_identity_float4x4, for: bearing)
let transformMatrix = simd_mul(rotationMatrix, translationMatrix)
return simd_mul(matrix, transformMatrix)
}
static func translationMatrix(with matrix: matrix_float4x4, for translation : vector_float4) -> matrix_float4x4 {
var matrix = matrix
matrix.columns.3 = translation
return matrix
}
static func rotateAroundY(with matrix: matrix_float4x4, for degrees: Float) -> matrix_float4x4 {
var matrix : matrix_float4x4 = matrix
matrix.columns.0.x = cos(degrees)
matrix.columns.0.z = -sin(degrees)
matrix.columns.2.x = sin(degrees)
matrix.columns.2.z = cos(degrees)
return matrix.inverse
}
}
To calculate the azimuth added the extension CLLocationCoordinate2D
extension CLLocationCoordinate2D {
func calculateBearing(to coordinate: CLLocationCoordinate2D) -> Double {
let a = sin(coordinate.longitude.toRadians() - longitude.toRadians()) * cos(coordinate.latitude.toRadians())
let b = cos(latitude.toRadians()) * sin(coordinate.latitude.toRadians()) - sin(latitude.toRadians()) * cos(coordinate.latitude.toRadians()) * cos(coordinate.longitude.toRadians() - longitude.toRadians())
return atan2(a, b)
}
func direction(to coordinate: CLLocationCoordinate2D) -> CLLocationDirection {
return self.calculateBearing(to: coordinate).toDegrees()
}
}
Problem 2. Excess AR objects
The next problem we encountered was a huge amount of AR objects. There are a lot of historical places and sights in our city, therefore dice with information merged and crawled one on top of the other. With great difficulty, the user would be able to make out part of the inscriptions, and this could make a repulsive impression. After conferring, we decided to limit the number of simultaneously displayed AR objects, leaving only points within a radius of 500 meters from the current location.
However, in some areas, the concentration of points was still too high. Therefore, to increase the visibility, we decided to use clustering. On the map screen, this feature is available by default due to the logic embedded in MapKit, but in AR mode it was necessary to implement it manually.
The basis of clustering was the distance from the current position to the target. Thus, if the point fell into the zone with a radius equal to half the distance between the user and the previous attraction from the list, it simply hid and was part of the cluster. As the user approached it, the distance decreased, and the radius of the cluster zone decreased accordingly, so the sights located nearby did not merge into clusters. To visually distinguish clusters from single points, we decided to change the marker color and display the number of objects in AR instead of the place name.
To ensure the interactivity of AR objects, a UITapGestureRecognizer was hung on the ARSCNView, and in the handler using the hitTest method we checked which of the SCNNode objects the user clicked on. If it turned out to be a photograph of a nearby attraction, the application opened the corresponding album in full screen mode.
Problem 3. Radar
During the implementation of the application, it was necessary to show the points on a small radar. In theory, there shouldn’t be any misunderstandings, because we already calculated the azimuth and distance to the point, even managed to convert them into 3D coordinates. It only remained to place the points in two-dimensional space on the screen.
In order not to reinvent the wheel, they turned to the Radar library , whose open source code was published on GitHub. The vivid previews and flexible settings of the example were encouraging, but in fact the points were shifted relative to the true location in space. After spending some time trying to correct the formulas, we turned to the less beautiful, but more reliable option described in the iPhone Augmented Reality Toolkit :
func place(dot: Dot) {
var y: CGFloat = 0.0
var x: CGFloat = 0.0
if degree < 0 {
degree += 360
}
let bearing = dot.bearing.toRadians()
let radius: CGFloat = 60.0 // radius of the radar view
if (bearing > 0 && bearing < .pi / 2) {
//the 1 quadrant of the radar
x = radius + CGFloat(cosf(Float((.pi / 2) - bearing)) * Float(dot.distance))
y = radius - CGFloat(sinf(Float((.pi / 2) - bearing)) * Float(dot.distance))
} else if (bearing > .pi / 2.0 && bearing < .pi) {
//the 2 quadrant of the radar
x = radius + CGFloat(cosf(Float(bearing - (.pi / 2))) * Float(dot.distance))
y = radius + CGFloat(sinf(Float(bearing - (.pi / 2))) * Float(dot.distance))
} else if (bearing > .pi && bearing < (3 * .pi / 2)) {
//the 3 quadrant of the radar
x = radius - CGFloat(cosf(Float((3 * .pi / 2) - bearing)) * Float(dot.distance))
y = radius + CGFloat(sinf(Float((3 * .pi / 2) - bearing)) * Float(dot.distance))
} else if (bearing > (3 * .pi / 2.0) && bearing < (2 * .pi)) {
//the 4 quadrant of the radar
x = radius - CGFloat(cosf(Float(bearing - (3 * .pi / 2))) * Float(dot.distance))
y = radius - CGFloat(sinf(Float(bearing - (3 * .pi / 2))) * Float(dot.distance))
} else if (bearing == 0) {
x = radius
y = radius - CGFloat(dot.distance)
} else if (bearing == .pi / 2) {
x = radius + CGFloat(dot.distance)
y = radius
} else if (bearing == .pi) {
x = radius
y = radius + CGFloat(dot.distance)
} else if (bearing == 3 * .pi / 2) {
x = radius - CGFloat(dot.distance)
y = radius
} else {
x = radius
y = radius - CGFloat(dot.distance)
}
let newPosition = CGPoint(x: x, y: y)
dot.layer.position = newPosition
Backend
It remains to solve the problem of storing dots and photos. For these purposes, it was decided to use Contentful, and in the current implementation of the project he completely suited us.
At the time of the development of the mobile application, all the backends were engaged in commercial projects, and contentful allowed to provide for several hours:
- mobile developer - convenient backend
- content manager - a convenient admin area for filling data
The similar implementation of the backend was initially used by the teams that participated in the hackathon (mentioned at the beginning of the article), which once again proves that things like hackathons allow you to escape from solving your urgent tasks on projects, make it possible to recreate and try something brand new.
Conclusion
It was very interesting to develop an
Simple, at first glance, the project required a lot of working hours to implement and refine the algorithms, despite the fact that we used the standard SDK from Apple.
We recently posted the application in the AppStore . Here's what it looks like at work.
So far, in our database there are points only for Taganrog, however, everyone can participate in the expansion of the “coverage area”.