Virtual thread support vs. Virtual thread-based framework

After the announcement of Project Loom and the release of Java 21, many Microservices frameworks declared support for Virtual threads. 

Helidon 4 is actually based on them! So, what is the difference between supporting Virtual threads and being based on them? We’ll discuss this in this post.

Threads in Java are just wrappers over OS threads. This fact makes them quite an expensive resource. You can’t make too much of them. It takes significant time to start a thread and to switch between them. So, we reuse threads in different Thread Pool Executors.

The nature of the typical tasks for serving Microservices is that there are many of them, they run in parallel, and they run pretty short, so they do not obstruct threads. Usually, requests are more than threads available. Running each request was not optimal, so the industry moved to a reactive paradigm and asynchronous processing.   

Most frameworks rely on Netty, which asynchronously manages connections to send data over them, thus providing good performance. Developers often should create asynchronous code to facilitate performance. And it is hard to write, debug and maintain.

When Project Loom emerged, many Microservices frameworks started experimenting with it by changing the regular Thread Pool Executors to Virtual Threads Pool Executors.

Or create a hybrid schema to offload some tasks to Virtual threads based on some markers (like annotations, for example)

This approach demonstrated some performance gain since there were more (virtual) threads available in the executor. But there was all this reactive/asynchronous overhead used for task management.

In Helidon we also experimented with this approach. From the early versions of Helidon 2 and 3 there was an option to enable Loom thread pool executor on Netty. But there was no such a big performance impact, and the heavy reactive hard-to-maintain programming model was preserved.

So we decided to remove it and create a new Web Server from scratch, which just creates a new virtual thread for each request and lets JVM manage those virtual threads. Everything is rewritten in the blocking paradigm!

This means that JVM will schedule each virtual thread with the request to the available carrier thread.

Whenever a blocking operation occurs, for example, IO or a network call, the JVM scheduler unmounts the blocked virtual thread, finds another virtual thread and mounts it to the free carrier thread. So, its resources will be reused.

As the blocking operations are over, the virtual thread will be scheduled to continue execution on the first. And this is the Major difference between “Virtual threads support” and being “based on Virtual threads”! 

As a “side effect”, the MicroProfile applications created with Helidon MP should run significantly faster since MicroProfile APIs are mainly blocking.

To summarise, with Helidon 4, you can enjoy creating high-performance applications using the blocking paradigm, creating code which is easy to maintain and debug! Check http://helidon.io, our blog https://medium.com/helidon and https://github.com/helidon-io/helidon/ for more details!

(first ever!)Drive-In conference: Java edition

This year is a different one. COVID has changed the world drastically. What seemed usual before is considered as something to avoid these days. Regular in-person meetings may even be prohibited. The most conferences in the world were postponed, moved online, or even cancelled. This was the destiny of jPrime 2020 as well – we have decided to cancel it, since we truly believe that for a community conference live interaction is one of the most important parts. We sincerely hope that humanity will find the cure for this awkward disease and quite soon everything will go to normal. …normal in terms of 2019, not the new “normal” we have right now… and in 2021 we’ll have our jPrime conference back

As a result, all of the community activities almost frizzed. All of the in–person meetups were cancelled. Some of them went online with a variable success. Some big (even payed) conferences like Joker and Jpoint moved totally online with quite big good feedback.

In BGJUG we were a little bit “hibernated”. After 5 really intence years of community meetups and conferences we gave ourselves a summer break.

This relax time helped us to recharge our batteries and get prepared for some new activities!

Earlier this year some of the isolation restrictions were removed and we could return to semi-normal life. One of this “semi-normal” activities became the Drive-In cinemas – often seen in the western world, but quite rare in Eastern Europe. Actually it is very good combination between safety and “real-life” experience. The spectators are sitting i their cars, the snacks and drinks are delivered on request, the sound is directly transmitted to the cars via FM radio.

Our colleagues from EPAM Bulgaria have tested the idea to use this Drive-In cinema to deliver talks, and everything went smoothly! So they offered to us to make a joint event specially for the Bulgarian Java User Group. This was an opportunity not to miss!

We have started the preparation in August and on 30th of September 2020 this event became a very successful reality!

i’ll be quite honest – all this preparation was thrilling! I missed it so much!

For this type of conf there are several restrictions. Since it is an open air, it should be dark outside. Fortunately in September it is getting dark earlier. We had to fit in the slot 19:30 – 22:30. We have decided to make 4 sort time slots each 30 min. long. We have also tried to test different presentation formats:

– live

-semi-live

– .. and prerecorded

We had two great local speakers who came specially for this event (one of them from another city). They have delivered their sessions live.

Together with Adam Bien we have made a live prerecording of his talk with a small community of listeners in a confcall. This brought the “live feeling” in his talk. It is great honor to have him at our Drive-In conf!

And our special virtual guest Dr. Venkat Subramaniam has prerecorded specially for us his talk!

We had pop-corn beer and hot–dogs available for free for our attendees! Huge thanks to EPAM for sponsoring everything!

Let us take a look how it all happened!

Some preparation:

Kick-ing of the conf!!! (I am nicely parked by the regulations)

Image
Image

With a good social distance!

Our first speaker Kostadin Golev presenting live about good coding practices!

Our attendees are carefully listening in their cars

Food and beverages available all the time!

Image

Our first experiment – a semi–live prerecorded talk by the great professional and speaker Adam Bien. I am happy there is some my voice recorded there and transmitted to the audience. It was really very “live experience”!

For the attendees who came without cars, there was a special lounge with a good social distancing. To be clear, it is allowed not to wear masks at open-air events in Bulgaria.

Our next speaker came Stefan Angelov from Plovdiv to share with us some knowledge about Java and Deep Learning!

And as our final speaker we had the honor to have Dr. Venkat Subramaniam, who delivered a wonderful talk about some new features in Java 15! The audience was thrilled!!!

Everything went soothly! We were fully booked and all the park places were taken! We are really happy that people came from different cities, not only Sofia! We had attendees from Burgas (~400 km), Plovdiv, Dobrich (~500km), Haskovo, Vidin, Stara Zagora, Veliko Tarnovo… Many of them made their long way specially for this event!

Image

But all the good thing come to an end. 22:30 sharp we have closed our first ever Java Drive-In conference. As some of or speakers were prerecorded, we are holding their signs, .. to make them semi–present 🙂 Huge thanks to our speakers!!

It was like a breath of fresh air! A great new experience with all the safety precautions! Finally the long events pause is over!

I have received a lot of requests like “When is the second edition?” “We want next such event ASAP!”. Even some some speakers wrote me “When is the CFP for the next edition?”. This makes us very enthusiastic!!!

Huge thanks to EPAM Bulgaria, who were the generous sponsor of this event! THANKS A LOT!

I have made some research.. looks like we are the first in the world to make a Java Drive–In conference! It is a great new “pioneer” feeling! It is priceless! We love this innovation!

By the way, we have invented a Drive-In way of applause! Just take a look how awesome it is!!!

(thanks to EPAM and BGJUG for the photos)

Thoughts on Work from home. Vol 2.

More or less about a month the world is on quarantine. I’m strictly following all of the government recommendations – we are self-isolated, no random walks outside, only for supplies, all our interactions with outer world are online.. or with my walkie talkie radio for short distance talks (great fun by the way, a strange warm “analog” feeling). I’m a lucky owner of a balcony looking to the south, so I even made some tan on my skin.

I have filled my day with tons of activities like learning new stuff, flat repair and upgrades, lots of reading and movies/concerts watching. 24 hours are not enough for me to complete all the tasks I’ve planned. But.. this is quite usual for me.

In my previous post I’ve shared the idea that, working from home is a skill that should be trained. It does not come by default to the most of us. It became part of our culture that every morning except weekends we get up early, dress up and go to work. We go there, because we need to interact with our teams to do our job. For IT these days, this may be the only reason for us to go to work. So, I’ve decided to try to understand why most of us are still doing this and not interacting from home, although we have all the technical tools to do this.

Let us for now not take into consideration this quite typical excuse – at home people can’t concentrate because of the family and children issues. Let us take the case that the worker can stay calm at home and has the same setup as at the office. Thus, we isolate only the interaction difference cases.

I’ve recently read an interested post somewhere in the social networks, that with this isolation the author of this post is missing live communication which include not only chat, voice and the information delivered via this voice, but also the body language, the partners look, his/her habits, even the smells. Yes, usual live interaction is a much more complex activity than just delivering some information. And of course, any remote interaction cuts off some of our senses, blocking these non-obvious information channels described above. As many people heavily rely on those perceptions, they feel disorientated. They can’t elaborate the trust to the partner. This my point was confirmed by one phycologist I have the honor to know for many years.

But as we continued this discussion, mainly discussing my case, when I’m stuck at home most of the time, and chats, voice and video calls are my only ways of interactions, why it is comfortable for me and so uncomfortable for the others. We have tried to search for some other jobs, where “being remote” is natural.

The first thing that came to my mind: pilot and ATC (air traffic controller) communication. It is always remote, it is over voice, and it is extremely effective. There is no need the pilot first to drink some beers with the ARC, find out if he/she can trust him/her and then fly. He doesn’t have to build trust, friendship etc. They just work with each other using common domain specific language. In some very rare cases they will be ever able to see each other.

If we take another aviation case – there is no more such thing as a permanent crew. For every flight, the captain and flight officers, as well as the cabin crew are randomly selected. Thus “human factor” mistakes and “unofficial hierarchies” are eliminated. Everybody in the crew knows his/her role and does it professionally. They don’t need to go get drunk together to be able to work effectively. The captain sure, that if his first flight officer is sitting by him, he is able to apply the commands and do the job.

I’ve come to realize, that somewhere deep inside, I try to use these techniques in my developer job. I have a limited perception communication channel (mail, chat, voice, video) and I’m focused not on my partners personality, but on their professional qualities (their code and the terms of the delivery). I sincerely do not care who’s on the other side – a girl or a boy, Russian or American, Christian or Muslim etc. I start my interaction with the full trust and expect the opposite side to be mutually sincere. And I can only judge and expect to be judged for the work done and tasks completed, not for how I like the way the partner is dressed. After all we are here to do job, not politics. If I see from the metrics we establish together, that the job is not done, I say – Bye! Sorry, nothing personal!

Still there are many activities requiring this full perception via all sensor channels like theatres, performances, conferences and concerts, politics – I’ve mentioned before, sports etc. Webinars will never replace conferences, movies will never replace theatres. Watching TV will never replace travel. They use different set of sense and perception mechanisms.

But does our IT job require all of these perception levels to be completed in the most effective way? I strongly believe that no. Even now we have all the necessary tools to interact. Moreover, we can even shrink this toolset to some async chats, and with some adjustments to our interaction “protocol” we can become even more effective and expand the geography of our workers to any place of the Globe.

Is it only IT? I would say no as well. Here my father is a good example, I learned a lot from him. He is an extremely qualified nuclear engineer. And his main instrument is his brain. He has organized his work to be almost totally from home, unless he is required to go onsite for maintenance activities. You may ask how is it possible? The communication channels and the tools he has are simply enough for him to do the job. The level of perception is just ok. So why lose time in transport and go to the office, when he can save this time for some more perception intensive activities, like theatre for example (hopefully soon again)?

The ability to fully utilize the communication channels with their level of perception is one of the keys to digital transformation happening now and accelerated in an unfortunate way by this COVID-19 situation. Many global players have understood this in the right time and that actually helped them to survive.

Yesterday evening I’ve performed several virtual flights on a sophisticated simulator. I didn’t know with whom I was actually playing with. Just their nicknames. I’ve declared that on this type of aircraft I am a rookie, and I’ve received a lot of support and we’ve completed our mission. We were using same domain specific subset of English with full trust and respect to each other. I’m very grateful to those guys and gals. So flightsim or an IDE? I don’t see difference. And this is the awesome future we’re living in!

Thoughts on Work from home

Disclaimer: These are only my personal thoughts, not pro tips. It is late in the evening, I’ve just finished watching “Perfect Sense” – wonderful movie with beautiful Eva Green and brilliant Ewan McGregor. It is a bit scary that something similar to the plot is happening right now. In real world. With this COVID. Strange thing is that when I’ve posted about it in FB, they have permanently removed this post. May be this movie is considered dangerous right now..

Everybody is self-isolating, all events worldwide are cancelled, borders are closed, airlines are close to bankrupt, my cable TV provider has opened all channels just to keep me home… Toilet paper has become a new currency… Or even a new religion.

On the other hand, I’m just sitting home and doing my job. As always. I’m coding, committing, drawing architectures, attending conf  calls, chatting, doing some extreme programming via shared desktop.. yes, I’m a software developer. I’ve been doing this for 13 years so far, and almost always from home. I have a reason for this – after a ski accident I’m using wheelchair for mobility. I simply can’t go to the office every day, due to tons of medical reasons, even though the office may be adopted. Every day I get up at 6 am to do my 3h exercises and medical procedures to be able to start work at 9 am. And in the evening, I have to do them again. Going to the office every day will simply kill my body.

Nevertheless, most of my career I’m employed, with few periods of freelance/startup activities.

How did work from home changed for all of these years. Actually, not much. Back in 2007 we had almost the same tools as today, may be slightly less powerful, and the Internet was a bit slower. When I started, we already had: VPN/extranet, voip/confcalls, various development and collaboration tools for distributed teams.. everything! Instead of slack we had icq, instead of zoom/jitsi/hangout we had webex… Many of those tools went to the browser and in the clouds, requiring no setup and allow instant usage.

The only thing that was different was the attitude. Most of my managers considered remote position as something extraordinary, a kind of a huge benefit for the employee. They did not trust the remote worker. They thought (and still many of them think) that “remotee” just plays games and does nothing. And only in the office, where there is full control, workers are pushed to their limits to deliver. Only if the team is collocated in an open space the team is able to be productive and efficient! The team shares ideas, can hear somebody asking someone for support, and join conversation to help. Theoretically this sounds awesome.. but in reality, this just leads to permanent distraction, constant search for meeting rooms, inability to concentrate on real work. There are a lot of scientific researches on the topic. Yes, programming is a tough brain activity. To write our code, most of the time we need to think and imagine! Well, if the company culture measures the quality of its work in “manageability”, not in “product deliverability” then yes – open space is the best option. Marissa Mayer even believed that getting all workers in the office could save Yahoo, but, you know this did not help at all.

As I already mentioned, not much has changed in technology. Back in those days we already had the toolset which was quite enough. Although nowadays we have much more remote workers, most of the IT positions still require full time office presence. Even though the worker will do absolutely the same stuff using absolutely the same laptop, still preferring asynchronous communication with his team mates via chat, trying to keep it quite in the open space. Why? To keep everything “manageable” or even “micromanageable”. It is still considered ok to burn tons of kerosene to fly to another country for a 40 min meeting. It is still considered ok to lose several hours every day just transport the body to the office and back.

But fortunately, not everywhere. I see two polar approaches: companies, that do not bind their workers to a specific collocated desk in the office, and those who do it. In some companies, employees can work from every spot of the office, or from every office worldwide. And there are many others (not so well known) which put their employees in a cubicle or just a desk and monitor their presence. Unfortunately, the last ones are still the majority.

But suddenly this COVID has arrived. Definitely a bad situation. Putting all of the workers in open space workers became a great risk. But let us leave the medical stuff to professionals. It is just bad to be collocated right now.

So, what do I observe: nothing has change actually for me. I still my job as usual, still connected, still delivering. But most of my “bound” employed friends and employees are experiencing some huge stress right now! They are both technically and mentally not prepared for what one of them has named – “the big separation”. All of a sudden, managers can’t micromanage and employees can’t be micromanaged! Most of them don’t even have the vpn/extranet and confcall set up. As a result, most of the employees are even forced to go to the office (something I consider as a crime during this dangerous period), or the delivery is just stopped, causing a lot of business losses.

On the other hand, the “not bound” employers (hey, I’ve invented my own vocabulary) have quite easily transformed their work to be “home office” based only. They are still able to deliver in a predictable manner; all of the communication and decision-making processes stay almost the same. It is only the physical location of the worker that has changed. Virtually all of them are there and do deliver!

What was a bit surprising for me is that my twitter has filled up with yammers like “Oh, I can’t work from home! It’s not the same experience!”, “I miss smalltalks by the cooler”, “I want my long dinners back! Where is my free food?”, “I want to play ping pong!”. Looks like the most of the developers are too much used to go to the office and are unable to organize their own self-discipline. As they were forced to move to home office many of them said – “Damn, I have a feeling a work a lot more now..”.

Work from home is not just a benefit, it is also is a developed skill. In my medical case I had no other option! But with this COVID situation, many of us will have to learn that. Will have to elaborate their code of conduct with people surrounding them, allowing and respecting their isolation. Will have to learn how to self-organize and gain discipline.

Fortunately or unfortunately the WFH skill may be more and more demanded by the market. Just because it is economically more effective – no need to pay office space, no need for enormous transportation expenses etc. This does not mean this will totally replace the offline job. It is like saying that television will replace theatres. This won’t happen. But this change in culture is happening right now, and this unfortunate COVID will accelerate it.

We’ve got a lot of examples of how people utilize this way of interaction for many years so far. Most of the open source projects are developed by distributed individuals in an asynchronous way. Gaming has almost totally gone online, allowing huge cooperative scenarios happening synchronously with people spread around the globe. Funny thing is that most of them won’t even meet offline, they don’t know their real names, whether it is a boy or a girl on the other side, the color of their skin, their religion, nationality etc.. But they work together as a team. Their “usefulness” is measured by their commitment to the common goal, not by their location and ability to gather in one room.

By the way, this situation may be a driver for many other professions to go online. Surgeons and even Air Traffic Controllers are already able to do their jobs remotely.

As for now, all of my previous employers, which some time ago have denied my participation in many projects because of my inability to be present at various offline meetings (meeting driven development) have filled up the Internet with great pro tips for how to work from home! I hope they really have changed, not just following the hype.

But.. I’m pretty much sure, after this COVID craziness is gone.. everything will go to the previous(ish) state.

And I’m now listening to Eurythmics song “sweet dreams”.

WebAssembly for Java developers

What is Webassembly

A lot of things have changed since the first web page was published. Back in the days it was a static text file with some html markup. For the last so few decades web pages evolved to some really thick clients allowing a lot of interaction and having a lot of functionality running in the browser on the client side. There was only one issue in this evolution – the only way to develop this functionality was using JavaScript. This scripting language, designed exactly 24 years ago, was not designed to be used to build such heavy or involved functionalities. As a result, today’s web faces a lot of performance issues and is really bound to one and only technology. Modern browsers sometimes do real magic to speed up the performance of web pages, but it is obvious, We should not try to work around the technology limitations by throwing more hardware, compute and memory resources at it. Every time we load a web page, we load HTML, CSS and JS data, we parse it, create AST, optimize it, compile or interpret it. It is a huge waste of resources.

Since it is really hard to break de-facto standards, W3C has created an extension technology called WebAssembly. We should be clear here – this is not a full replacement of JavaScript, but an extension, focused on high performance.

So, according to Wikipedia, WebAssembly (often shortened to Wasm) is an open standard that defines a portable binary code format for executable programs, and a corresponding textual assembly language, as well as interfaces for facilitating interactions between such programs and their host environment. The main goal of WebAssembly is to enable high performance applications on web pages, but the format is designed to be executed and integrated in other environments as well.

Wasm code (binary or bytecode) is intended to be run on a portable virtual stack machine (VM). The VM is designed to be fast, to parse and execute than JavaScript and to have a compact code representation. A Wasm program is designed to be a separate module containing collection of various wasm-defined value and program types definitions expressed either in binary or textual format (that both have a common structure).

Except in a virtual machine, wasm can also be compiled to native binary (with 3rd party compilers), and this is an easy way to release something that is not usually native as native, as long as it doesn’t require heavy runtime libraries.

WebAssembly is a fairly new technology and its development is in progress. But there already is an MVP (Minimal Viable Product) implemented in the most of the browsers and available out of the box. But, to make it really worldwide standard, we need some more time!

There have been around 40 programming languages reported to support Wasm as a compilation target. Java and Kotlin are among them!

https://hacks.mozilla.org/2017/02/a-crash-course-in-assembly/

Various tutorials are available in YouTube and other platforms for better understanding WebAssembly. It is strongly recommended to get more acquainted with WebAssembly before reading this article.

What about Java?

Let us try to create a WebAssembly from Java code. For this we have several options.

One of them is JWebAssebmly, which is is a Java bytecode to WebAssembly compiler. It uses Java class files as input. That it can compile any language that compile to Java bytecode like Clojure, Groovy, JRuby, Jython, Kotlin and Scala. As output it generates the binary format (.wasm file) or the text format (.wat file). The target is to run Java natively in the browser with WebAssembly.

To export a Java function to make it accessible from JavaScript, you must add the annotation de.inetsoftware.jwebassembly.api.annotation.Export.

As a result the following wasm will be generated (text representation):

To import a JavaScript function to make it accessible from Java, you must add the annotation de.inetsoftware.jwebassembly.api.annotation.Import. The method can be declared native or can have a Java implementation which will be ignored on compiling.

The project is very young. It is only version 0.1 so there are a lot of limitations.

Another option will be TeaVM.

We can set it up as s regular maven/gradle dependency:

Write some Java code of the stuff we want to export as a Wasm binary. @Export annotation tell us that this function will be visible from JavaScript

With a maven plugin we specify where to export the generated WebAssemblies (take a look at the TargetType):

Then just prepare the html file:

And the JS file:

And we are done! Now we can code in Java, compile it to wasm, and run it on the client side!

You can play with this code here https://github.com/dalexandrov/WebAssemblyFromJava

How about Kotlin?

They did it their own way. Since Kotlin has an experimental project Kotlin/Native, which has the compilation target of LLVM, it serves as a base to generate wasm binary out of it.

We should only specify a build target of “wasm32” in our gradle build script and as a result, on the output, we should get wasm binaries and a JavaScript wrapper to work with this assembly.

This project is still experimental and there are a lot of thing to be done. Unfortunately, the documentation is also “in progress”, so you will need a little hacking to play with it. But it is a lot of fun!

Can we use Webassembly from Java?

On the other hand, Java can also “consume” WebAssembly, or to be more precise GraalVM can do this.

In his article Aleksander Prokopec has announced that GraalWasm currently implements the WebAssembly MVP. Further reads here: https://medium.com/graalvm/announcing-graalwasm-a-webassembly-engine-in-graalvm-25cd0400a7f2

As a conclusion..

As a wrap up, we can say that WebAssembly is a very young, but a very powerful technology. It is still in MVP stage, with a lot of specs yet to be implemented (like multithreading, SIMD, etc).

But even now WebAssembly can run with speeds up to 1.2x native. And even now we can offload many server-side activities to the client’s browser and avoid performance issues. And we are able to do this from Java world.

Additional resources:

My talk about Webassembly and Java https://www.youtube.com/watch?v=93z9SaLQVVw

Further reading about Wasm and graalVM https://github.com/neomatrix369/awesome-graal#wasm

Thoughts on microservices: Throwawayware.

Картинки по запросу throw away this

Microservices.. yes, nowadays they are everywhere! In my previous post I’ve shared my thoughts that sometimes just a monolith can do the required task without being cut into microservices. But it is wonderful if microservices are the most suitable solution! This is definitely lovely!

First, we’ve got a lot of benefit it provides to us, second, we’ve got a great technology support to implement them – there are a lot of frameworks and platforms helping us develop some really good microservices.

The most of the conference talks I have recently seen are basically “Hello world” talks. The main message is “Hey, look how easy it is to create a microservice!”, “Just few annotations, and you are up and running!”. How great this is!

But when it comes to real life, most usually between start of the project till the first really useful microservice there is at least one sprint. More often three sprints. You’ll say – “That is ok! We are serious people doing serious enterprise!”. I still ask myself – “What can be so huge in one microservice, that it may require several weeks to be created?”. “Why then this service is called MICRO?”

I’m not sure if there is an official definition of a microservice, but a well-established idea is that it is a piece of software which is designed to perform only one single function, but in the best possible way. It should communicate with the rest of the world with the most lightweight protocol possible, like REST. So, if we have a login functionality, a microservice should be able to login a user in the most fast, easy, reliable, secure way. Only login and nothing else. For logout there should be another service. The size and complexity of such a service are not in scope of micro, they can be of any scale. Technically, there should be only one function exposed, but what happens under the curtains are just implementation detalils.

This is why, the most of such services become really complicated, often even overengineered. From my experience, at the end of the day, each microservice in enterprise webapps becomes a usual three-tier application. It has its “UI” simplified to REST endpoints, it has its business layer with several services interfaces and their implementations, an infrastructure layer to talk to DB and other services. The interlayer communication is done via value objects. As a result, a usual call of a function of this microservice ends up with several data coping from one value object to another (in one of such microservices I’ve seen up to 14 of such hops). Why would you do this? The typical answer is if some part of the service changes, we will change only this part of it. Fair enough. If in return I ask, how many times in your life have you ever changed one of the tiers without affecting the others? Тhe answer is usually – never.

Once again, nothing wrong here. The complexity of the implementation of a microservice is orthogonal to the interface it provides.

Another question: how many times have you ever received a fully functional “freezed” interface definition? The spec that never changes? A guaranteed never to change document? In my case – never!

The changes are coming constant or even increasing speed. Even in terms of one API version. You may say: You are doing it wrong! You should plan more carefully, you should rethink your spec versioning policy etc. And you will be completely wright!

But.. there is something called real life! There are business needs that require quick reaction, quick change, quick time to market!

Looks like this constant change is just inevitable. For my 13 years in software development there were practically no projects that had really smooth stable development. A least every sprint there were some “tiny” spec changes, causing all codebase to suddenly go red. I’m sure I’m not the only “lucky” having this.

How should we solve this kind of problems in the microservices world? Its quite time consuming to create beautiful, complex, nicely architectured, (at least) three-tier microservices without frozen spec. As I already said, those spec changes usually require changes in all of the layers. This is hard! Most of us thought that’s just the way it is. Software development is hard.

But this situation kept me bothering. May be there is a way to expedite this change reaction? May be a microservice should be micro in all aspects? I mean really tiny? With as less internal abstractions as possible? Really tight coupled inside? When I’m writing this, I have a feeling I’m breaking the law. It is like I’m cancelling everything I’ve learned from the CS courses in the university. Why don’t we throw away all of the … tiers? Make it a really tiny one-tied micro monolith? A microlith? So that the object or even a direct line with data we receive from the infrastructure we transform to JSON object in one single class? With all the business logic also included in this one and the same class? You are absolutely eligible to say – “are you nuts?” What if the infrastructure changes? What if the Rest API changes? What if the business logic changes? You have to rewrite everything!

And then I suddenly say – “Yes! We will rewrite this service from scratch!”. Luckily, a lot of code in the service can be just generated. So, yes. We will rewrite it from scratch! We copy/paste. We do all the possible anti patterns. Only to make it work according to the spec and pass the tests! “But isn’t it a lot of effort?”. My answer is – “Pretty much the same as changing all of the classes in all of the three (or more) tiers!”. And people usually say – “Hmm…”.

This may sound really strange, but through several last projects, writing some really ugly small one tier microservices with really almost no architecture and no refactoring, then rewriting this from scratch saved me a lot of efforts. I have even created a name for this piece of software – “Throwawayware!”. Try to say it quick!

The next question that usually comes is – “Do you put this ugly thing in production??”.

My answer is: – “Ok, you got me! No, this does not go to production! May be sometimes..”. What actually goes there?

Putting this ugly little thing in production will be just catastrophic. That’s why I’ve tried some mixed approach.

First of all, we usually develop the tests to follow the contract. Yep, although I’m not a big fan of TDD, I think this is a good place to use it. Adjusting the tests to follow the spec and the contract is always the first priority to us!

Then we usually have 4 to 10 iterations on each microservice in this ridiculous “throw away” way. Funny thing is that up to 50% of the services do not even survive through those iterations! At some point the service may come obsolete even before production. This means that we usually save a lot of efforts for not creating something complex for something that will never be used (“Bingo!”).

Then when the service gets through those several iterations, we consider this service as “survivor”. At this time usually, the spec stabiles as well. And we do the same for each API version.

Just like the Garbage collector in JVM.. Haha! Yep, like the GC! Lovely!

For the survivors we usually make some really complex refactoring, or rewriting it completely new. We try to make their code really readable, expandable and maintainable. Quite often from the usage of those “throw away” implementations we can see some before unpredicted usage cases, like for example the pressure it should hold, or security, or fault times etc. This may end in three-tier architecture, but with really nicely designed layers.

Looks like this “two stages approach” works really well! Funny thing is that it fits perfectly in our SCRUM cycle. May be, I’m wrong, but this approach saved us a lot of effots and helped us establish good quality implementations in shorter time. In three projects so far.. We’ll see how it goes!

Any critic is welcome though…

TRIP report: Codefest. The siberian adventure. 🇷🇺

Right after Joker 2018 I was approached by one of the speakers Ivan Ugliansky with a very interesting proposal – “Hey, you know I’m from Novosibirsk, we have quite a big conference named CodeFest happening usually in the end of March. Would you like to come to Siberia?”. I answered – “This will be an honor for me!”.

It took me about 12 hours to reach Novosibirsk from Sofia. First 3h30m from Sofia to Moscow. Then I had my shortest night for this year – we took off at 21:30 MSK and landed 5:30 NSK time. The time passed quickly, the Aeroflot flight attendants were we so beautiful, that all of my attention mainly focused on them.

When we landed in Novosibirsk, I just couldn’t stop staring at my GPS position. Wow, I’m almost in the middle of Eurasia!

No photo description available.

The first day was quite intense! My great friend Ivan has made a wonderful excursion to a place named “Akademgorod”. In English this may sound like “academics village”. It was created from scratch bak in the 60s to accommodate Soviet workers of science in a very peaceful close to nature environments. As a result it has transformed in a huge scientific cluster. A lot of institutes and research laboratories are now located there. There is also a huge university.

Students and specialists from all over Siberia and even Russia come to this place for study and cool projects!

No surprise, there is a big IT presence here. A lot of Russian and not only Russian companies have their R&D here, and no surprise the CodeFest conference gathers more than 3k attendees.

I had the privilege to be invited to the office of JetBrains

and the office of ExcelsiorJET

I had some truly wonderful time there! We had some pizza and great chat! Huge thanks to Ivan for being such a great host.

The end of the day was dedicated to a special event – the local JUG meetup. Together with Paul Finkelstein I had the honor to give my talk about Java and JPU in front of the local community. The event took place on the 22nd floor in the office of the famous Russian company 2GIS. The location was really awesome!

More than 150 people come to the event! It was fully packed with even standing people!

To be honest, I was a bit scared to be in front of these people. Siberian developers are famous to be the most hardcore developers in the world. Mostly the strongest compiler writers. Isn’t my talk going to be too smoothy for them? At the end of it I was happy to find out that it was really ok! I had a lot of questions afterwards!

It was a sincere pleasure to talk to this community!

Картинки по запросу novosibirsk expo center

The next day was the first day of the CodeFest. Luckily I had my talk scheduled for the day two. It was really good news for me, since I still had some jetlag.

So.. the conference is huge! It has 7 tracks and it is not focused only on Java.

*Siberia is here!!

Actually the variaty of technologies and a SC fields was a big plus for me. I learned a lot of new stuff. I think I was mostly stuck at the Data Science track. I’ve been playing a lot with tensorflow lately, so it was a natural choice for me.

I’ve heard some nice Project management and Team lead oriented talks as well.

I was even interview on the conference radio. I had a lovely chat with Vladimir Plizga from CFT.

The conf is huge!

The day one ended in the local pub with the concert of … the speakers! Singing about.. coding!! Awesome experience!

My talk was scheduled to be the first on day two.. I’ll be honest, I have expected no more than three people to show up. Three listeners would have been a victory for me!

Surprisingly.. it was almost packed for my Microprofile.io talk! (based on the MicroProfile.io tutorial created together with Ivan. St. Ivanov)

Every went smoothly! All of my demos worked! What a releaf!

Usually its not just giving a talk on CodeFest and you are free. After the talk there is an expert zone, where the speaker can make some live discussions and answer audience questions. I spent almost an hour there! I’m really happy that Microprofile.io enjoys so much interest! I hope it was useful for the audience!

The second part of the day I have decided to spend in the city exploring it.

The famous Transsib railway is crossing the big Siberian river Ob here.

The old and new come together!

The cultural life is very intense!

The famous Opera and Ballet theatre is really huge!

Siberia and Novosibirsk are definitely wonderful places to visit! If you have the chance to go there, you definitely should!

I’ve met really wonderful people and had some amazing time!

The next day I was in the airport having something I’ve almost missed – Siberian pelmeni!

Image may contain: food

Ok, I’m going home! Siberia, you are endless and beautiful!

Image may contain: sky, airplane, cloud, outdoor and nature

Yet another 12000 km up in the air! (SOF-SVO-OVB-SVO-SOF)

Huge thanks to Ivan Ugliansky for making this possible.

Quarkus: a blast from the future

Yep, Quarqus has definitely exploded just few weeks ago. As one of my friends said, it was marketing lvl8080! Just listen to the slogan – “Supersonic Subatomic Java!”. “A Kubernetes Native Java stack tailored for GraalVM & OpenJDK HotSpot, crafted from the best of breed Java libraries and standards”. Damn, so many buzzwords in one place! The twitter literally exploded as well. Literally everybody twitted about Quarkus.

I said – nah! Yet another microservices framework. There are many available now. The choice is really big. So, I almost ignored it. I’m honest here. Still the last Sunday I’ve downloaded it and played with some examples. My first experience was a failed build. I’ve submitted an issue in GitHub, and it turned to be my local maven issue. The fix was just to upgrade it. But as result this issue opened a discussion, and very same Sunday a maven version enforcer was added. The power of OSS! There are no weekend in OSS! Kudos to the team!

The twit flow nevertheless continued to appear in my feed. This somehow forced to open the Quarkus website and explore more. There was one logo that grabbed my attention:

I said – mmm MicroProfile. Lovin’ it! So it looks like I may try to pack a microservice (which is not Spring) in a GraalVM native image.

Ok, I need to try! This thought came to my mind about 3 a.m. this Thursday. Well, technically it was Friday already…

Luckily enough about a month ago we have deployed some CSV aggregation service. Yet another stage of some fancy bloody enterprise infinite process to generate some invoices. We have made it the most modern way using only MicroProfile specs, on WildFly server, running in some OpenShift pods (yes, we have a lot of RedHat licences). Mmm, a loot of cool buzzwords as well!

The service itself was completely stateless. With three rest clients we gather some data from there other services, mash them up in some CSV and serve it as a plain text on rest request! Technically it is insanely stupid! … making it the perfect candidate to be tested with Quarkus. By the way, all of the pipeline is still under development, so thus we’re not going to ruin someone’s bill and invoice.

Still the service has to work under some pressure. Thus OpenShift scales by adding pods. Since the service is stateless, we are totally ok with that! We also don’t have to care about rest providers from the other side. It’s their responsibility to scale 🙂

So, now comes the fun part!!! I’ve started a new project by copying some of the Quarkus quick–starters. Yes, I can generate a pom from an architype.. but I’m lazy.

I then just copy-pasted the code from my old project.. and it shined red.. Especially the the MicroProfile annotations.. ah. What’s wrong? 3 minutes of googling told me that for the MicroProfile stuff I need to add some Smallrye Quarkus extentions.. Ok, copy–pasted in the pom.xml.. Yay, the red is gone.

mvn clean package

… and BUILD SUCCESSFUL!

Ok, you made me sweat! Now let us:

mvn quarkus:dev

.. and .. http://localhost:8080/api/v1/invoicingRXT/1253211 (some contract)

AND I HAVE THE RESULT! That’s a WOW!!! This was one the fastest migrations I’ve ever done! I LOVE STANDARDS! Long live MicroProfile!!!

Ok, I have earned my coffee. It took me.. 17 minutes. Yes, this was mostly copy–paste, but look how cool is that!

… But, thats not over! Quarkus documentation says if I add -Pnative to the maven build command it’ll produce a native image! Let’s do it! Luckily I had already setup the GraalVM installation.

mvn clean package -Pnative

Waiting.. 1 minute… 2 minutes.. BUILD SUCCESSFUL! Lovely! Yes, it takes some time to build an image! I should admit, that the business logic code was written in quite straightforward way, without any fancy constructions, just some POJOs. There were no GraalVM specific issues.

Now let us just run the executable:

./invoicingRTXApp-runner

and the service is up and running in about a second! Although the console says the startup time is 0,212 sec. Technically from the run command to running service is about 2 seconds. THAT DOESN’T COMPARE to ~49 seconds startup time of the WildFly server..

Ok, now lets go to http://localhost:8080/api/v1/invoicingRXT/1253211 and what I see:

“”,””,””,””,””

Something went wrong!!! Why the example from the quick-starters work? Hmm… It looks like I’m missing one annotation over some classes – @RegisterForReflection. Reflection works a little different in GraalVM. Building it once again. Waiting another two minutes. Oh, how can two minutes can last so long..

Good! BUILD SUCCESSFUL! Now lets go to http://localhost:8080/api/v1/invoicingRXT/1253211 and what I see:

data1,data2,data3

IT WORKED!!! (Now imagine the famous Dexter from Dexter’s lab cartoon shouting this loud). That is so damn cool!!!

Картинки по запросу dexters lab it works

Nice! It’s been 42 minutes since the beginning of the experiment (coffee break included)!

Ok, now let us go back to the OpenShift setup. It will be nice to see if it’s ok in our test environments under some pressure. After making some yaml permutations I’ve rerouted 10% of the traffic to go the new native image pod. After 4 hours watching it work I see no errors! Only sometimes I receive some messages from surprised testers like “Something’s wrong, some pods start in 3 seconds only…”. And I say “Haha! This is magic.”

Now some intermediate conclusions:

– WildFly 78 Mb image, Native 21 Mb image;

– WildFly ~49 seconds start-up, Native ~2 seconds start-up time;

– WildFly ~300 ms per request, Native ~270 ms per request (we are dependent on other services).

– Wildfly ~359 Mb RAM (serving 1 request), Native ~22 Mb RAM (serving 1 request)

Lovely! Just, Lovely!

What can I say – Quarkus is definitely a good thing to play with. It showed extremely well in (almost) production code. Yes, I know the example is really stupid, but that’s a real world demand. The migration took me about 40 minutes by just copy–paste. I love standards!

The coming Monday I’ll make a full native setup, we’ll see how it goes! May be even in production soon..

Disclaimer: Sorry, I can’t share the code. It’s corporate..

Trip Report: DevoxxUA 🇺🇦

Картинки по запросу devoxx ukraine

In the beginning of this year I was quite skeptical, that I will make more then 2 conference, including the one I’m part of – jPrime.

But the end of the year was definitely very intense. After coming from St. Petersburg I had the pleasure to speak at several local events, including the oldest and most prominent local conference OpenFest. It was a great opportunity to meet not only Java programmers, but also to talk about different platforms and languages as well.

As I was in Oslo and St.  Petersburg Ukrainian guys Andrii Rodionov and Oleg Dokuka approached me and invited me to submit a talk to the inaugural Devoxx Ukraine. So I did it, and very soon I’ve received an email saying I was accepted! Actually I was quite happy, since I haven’t been to Kiev for more than 5 years already.

The girls from the organisers helped me a lot to get all the invitations and was able to enter Ukraine without any problems!

By the way, that was the first time I had a direct flight to a conference, without any layovers. This is so much great! I’ve started appreciating this fact!

The conference takes place in a beautiful congress centre “Parkoviy”,

with a beautiful view of the Dnipro river. Well.. at this moment it was a foggy day (like the most of the days in November in Kiev)

The venue itself is very nice and modern..

There was even a car parked inside!

The entrance (or I would say – “the portal”) was extremely cool!

Actually the theme  of the conference was like “You are in a spaceship!” which is really lovely.

Unfortunately I was there only for the second day. I’ve missed some cool sessions like the keynote by  the famous scientist Lawrence Krauss.  It’s also sad, that I’ve missed the BOFs at the end of the day, I’ve started enjoying them a lot. I like this live discussion that happens in between.

But what I definitely really enjoyed was the first session of the day two – “Software Engineering Theory vs. Practice: A Song of Ice and TireFire” by Baruch Sadogursky and Evgeny Borisov.

It’s not very much technically oriented, but the topic about the “Technological maturity”  of a company is something I really have to dive in. I often find myself asking why we need this or another technology to solve current problem. Is the technology we’ve chosen able to solve this problem in the most efficient way and are our teams ready to implement it. Definitely a good talk!

Now I’m looking forward to see the videos of the talks I was not able to see. There were a lot of them running in parallel, the choice was hard!

As for my talk… the was a challenge for me: I had to talk in parallel with awesome Orkhan Gasimov, Doug Stevenson, Nakul Mishra, Reza Shiftehfar,  Mary Grygleski .. and Dr. Venkat Subramaniam. But I was sincerely surprised when I had my room completely packed with extra chair and people standing..

So I’m very grateful for all people that have attended the talk! I hope that the attendees will find good application of the technology in their projects!

I’m happy to have received this awesome comment!

And as always – conference is meeting great people! And finally meeting them in person!

Like Michael Simons! Finally we have talked to each other in real life!

And Rafael Benevides with Dr. Venkat Subraniam

And many many more!!

As I was late to the lunch after my talk, I was arranged a special very Ukranian lunch made of borsch, vareniki and salo!

This definitely made this event very special for me! I’m really grateful to Alina and Irina for arranging it for me!

It was a wonderful conf. Made with all of the Devoxx traditions!

I had a lovely stay! I’m very grateful to everybody who made it possible!

Bye bye, Kiev! Till the next time I hope!

 

Trip Report: Joker 2018 🇷🇺

Image result for jokerconf

It will be a bit strange to call this blog post a “Trip Report”, bearing in mind that I’m fairly local. Yep, technically I’m living in two homes – St. Petersburg, Russia and Sofia, Bulgaria. So coming to Joker was kinda switching home, but to make it more unified I’ll still call this post a “Trip Report”. A trip report to Joker Conference!

There is one disclaimer I have to make: in this post I do not follow any timeline. Here I just want to share some thoughts and emotions.

So, what is Joker Conference? This is Russia’s biggest Java event, which usually takes place at October/November  in St. Petersburg.  The city itself is now unofficially considered as a technological hub, accommodating a lot of IT offices and foreign representations. Like Oracle for example, they used to have a big R&D office in it. You may be surprised, but Java ME for example, or the biggest part of lambdas in Java 8 were created in St. Petersburg. Now, for some reasons, there is no more R&D, but some other activities. Still a lot of banks, telecoms, retailers, aerospace etc have some quite big R&D offices in St. Petersburg. The Universities produce a lot of very qualified developers to satisfy the huge needs for that.

So, as a result Joker is a big conference. There are usually about 1500 attendees.

The conference is run by JUGru group, and it is absolutely professionally organised.By professionally I mean that conference organisation is their main job. Joker was the first one guys&gals have made, now there are more than 15 conferences they manage. And they  are not only Java oriented.

This was my fourth Joker Conference, and I kind of see its evolution.

For me it all started back in 2015 when it was held in one of the hotels and it was sincerely crowded, but starting from 2016 the conference moved to the Expoforum – a huge expo center near the Pulkovo airport.  This venue definitely has opened some new possibilities like convenient transforming auditoriums, big expo and dining area. The venue is really one of the best I’ve seen in my vast “conference junkie” career.

The stage is cool!

And there is a lot of attendees!

The conference has mixed Russian/English set of sessions, so the speakers were also mixed local and foreign. The audience is mainly local. By local I mean Russian/Belorussian/Ukrainian/Kazahk – or in other words Russian–speaking. But the quantity of foreign speakers grows constantly trough the years, and I consider this as a good sign. All the visa barriers are handled gracefully by the JUGru team.

Russian conferences tend to focus on more hardcore topics. Just introduction talks receive the lowest scores. Russians prefer hardcore stuff! Yes, the talk should be hardcore, with a lot of deep dives, algorithms and case studies. If you can describe, for example, why your JVM consumes few bytes (or gigabytes) more than expected, and you know what experimental flag to put in the JVM params to stop it from doing this, describing in implementation details why this actually happens – this will be a cool talk! You may say, oh, isn’t it kinda JVMLS? Not exactly, dissecting Spring Boot or some Micronaut or Microprofile stuff is the expected topic. Not just knowing that the framework/technology can do this or that, but how actually it does it, and what are the corner cases causing unexpected bugs.

Yes, hardcore is highly tolerated! Why am I saying this? Actually I know this from the source. This year I had the honor and privilege to be invited as a member of the program committee. We had a wonderful team of eight – Gleb Smirnov, Victor Polischuk, Andrey Kogun, Oleg Anastasiev, Vladimir Sitnikov, Ivan Krylov, myself and first Valentina then Daria, to select the best possible talks for this conference. So, we were able to preview all the submissions, make rehearsals, make corrections in the talks, sometimes even redo the talks from scratch. Yes, you’ve got me right. On this conference (or actually on all JUGru conferences) the committee actively works with all the submissions and the speakers themselves. If you submit your talk to Joker – be prepared, that in few days someone from the committee will contact you and ask you for a rehearsal.

Many foreign developers  find this weird. They usually say: “Hey, hey man, I gave my talks at JavaOne, Devoxx etc. I had my rooms packed. Why should I do a rehearsal?”. But at the end of the day, those who accepted to rehearse usually say: “Damn, that was so much useful! And the feedback you gave me is so valuable! And that was totally for free!”.

As a result – the content is very strong and nicely prepared!

This is one of the key aspects that makes this conference kind of unique – a very intense content preparation. Usually the committee starts its work half a year before the conference, and active rehearsals are made until almost the last day before the conference.

By the way, there was even special time slot for direct interaction with the program committee.

Image may contain: one or more people, people sitting and text

Another thing I’ve seen only on Joker are the discussion zones. Every speaker, after giving his talk, is usually invited to answer questions about his talk or continue the discussion. This usually lasts for an hour.

Image may contain: 2 people, people standing

Image may contain: 2 people, people standing

In case of Dr. Venkat Subramaniam and Josh Long those discussion zones have lasted up to three hours. People had so many questions, and they both were answering them all that time!

This year’s conference continues its growth not only in terms of quantity of attendees, but also in terms of activities. There were two additional stages set in the expo area, so that the companies a the booths are able to present some quickes at the breaks.

The booths were also full cool stuff.

Image may contain: 5 people, people on stage

Image may contain: 1 person, crowd and indoor

Btw,  unlike other conferences, it’s not easy to win a prize at these booths. To do this, you have to solve several weird programming problems, most of which are like puzzlers.

Image may contain: 6 people, people smiling

There were also so called experts areas set, where every attendee was able to ask the strangest questions  about certain technology to the coolest programmers and solve their concrete issue! .. Or ask them to sign a book!

Image may contain: 7 people, people sitting

Everything was done to keep everybody busy till the late night (literally) the day one ended up with several cool BoFs. Together with Shelaev, Pangin and Chuyko we have hosted one of them regarding moving to Java 11.. And it was pretty cool! We had a very good discussion, with a lot of interaction, … and we were forcibly stopped, since this could have run all night long! I’m not totally sure whether the Java 11 was considered as a game changer, but one day we will have to migrate to it. The main discussion about it was, as expected, should we pay starting from now???

Image may contain: one or more people, shoes and crowd

Image may contain: 11 people, people sitting, crowd and outdoor

The activities at the conference hadn’t been strictly about knowledge but fun too, like a big exhibition of retro computers and games everyone could try playing.

As always the greatest part on each conference is meeting great people! I’m definitely happy I could meet Sebastien Blanc.

So much great to meet Gerrit Grunwald! I think the last time we have met was in SFO in 2015!

Together with the coolest Tagir Valeev watching Andrey Pangin’s talk.

With the so cool JBaruch (as a JFrog)!

It was great to meet with Ivan Uglyanslki from Novosibirsk!

A great honor for we was to meet in person Robert Scholte – one of the maven creators! It was also great to discuss the future of maven with him!

And we all had a lot of fun!

I liked a lot talking to John McClean not only about Functional programming but about Irish music as well.

https://twitter.com/DataDuke/status/1053722000223596549

This is the most OMG picture I’ve ever taken:

It was very interesting to talk with foreign speakers not just about the technology, but also about Russia. The most of the speakers are for the first time in the country, and it is very delightful to hear that they like the town of St. Petersburg a lot! Some of them have said: “They show us different thing on the TV..”

And by the way (yes, yet another by the way), on this conference I was not only part of the committee, but also a speaker. I gave a talk about Microprofile.io based on the tutorial we have done together with Ivan St. Ivanov from our BGJUG. My session was scheduled as last of the day one.  Surprisingly the room for 700 people was almost packed! Never thought so many people would like to dedicate their time on Friday evening for the ” bloody enterprise”.

Image may contain: 1 person, eyeglasses

I would like to say huge thanks everybody for coming! I hope this will be useful for your next projects! I liked the phrase several people have said – “Oh, this microprofile is like the light in the tunnel for the enterprise! This is great! It is so cool the technology is evolving so fast! And the community is there to help!”

At the end of the second day, as a final nontechnical keynote we all have listened to a very both interesting and strange talk about people’s digital mental health by famous in Russia Dr. Kurpatov.

Image may contain: 1 person

Actually there were a lot of take-aways for me from this talk. I still keep training my personal multithreading, but at the same time I train the concentration on a particular important task I have to finish on time with great responsibility.

And the last but lot least thing I should mention – just before the conference, the previous day the world’s greatest java speaker Dr. Venkat Subramaniam gave  his wonderful two (technically almost three) hours jug session. I was very happy to meet him after he came to jPrime to Bulgaria earlier this year. It was absolutely great to have his book signed for me! I’m feeling blessed now!

fbt

As expected, it was fully packed on this event!

Just to mention that this event took place at the Oracle office in St. Petersburg. Like every other building there it looks like a palace!

Image result for келлерман центр

 

So, Joker was great! As always great! Getting more and more great!

Thanks Andrey!

Thank you Alexey!

And thanks to the team! It was huge honor and pleasure to be part of this event! Hope to see you next year!

Image may contain: one or more people and crowd

Cheers!

And St. Petersburg is just amazing in autumn…

Image may contain: tree, outdoor and nature

See you soon, beautiful home!

Image may contain: sky, cloud and outdoor