Posts in JAVASCRIPT

I had an interesting architectural/process challenge recently. A lot of the projects I've built lately have been for supply chain tools and analytics. I have to collaborate very closely with these supply chain teams around how their data is structured and accessed in order to build these analytics tools to their needs.

The most involved part of that collaboration is with their data analysts. They are excellent with SQL and they know their data structures very well, which is great because I can be more hands-off with business logic in my role. However, they like to frequently iterate on their analytics process, and their data needs to be scrubbed semi-regularly. This results in frequent adjustments to the SQL that hydrates the analytics tools I build. Manually rebuilding, retesting, redploying, and change managing the application on this frequent basis can be time consuming.

We decided to completely separate out the SQL code from the analytics tool code as it's own code repository. I did some quick training with the data analysts to get them set up with a code editor of their choice and an easy Git GUI so they could take charge of all the SQL logic that hydrates the analytics tool. When the analytics tool starts up on the server, it goes to look for those SQL files from the supply chain team's repo. It loads in those files on startup and keeps them available in the application runtime. Anytime the supply chain analysts make an adjustment to an SQL file, the pipeline that deploys the SQL code also triggers a quick restart on the analytics tool. So the tool always has the latest SQL to work with. There's a utility function on startup that validates each SQL file to make sure it's not using certain restricted operations, and fulfills certain minimum data set attributes.

This architecture allows them to make changes and deployments on a much smaller codebase without having to involve development team resources for our change process. They get to iterate on the data as quickly as they like and we get to be hands-off with business logic, so it's a win-win. This process has been working well for us for several months now.

This post continues describing a presentation I gave for my development team on Neural Networks and TensorFlow. Please read Part 1 first.

The second use-case I created was a model for simple plain-language search prediction. The idea I had was that a user could use a search input for the sales allowance reporting tool my team had built. The tool shows users the changes in sales allowances on ad campaigns, and it outputs those reports in a variety of formats depending on what the user needs. A language model might help users find the report type they want.

I had the team send me many different terms and phrases that users might use to find reports in the tool. I compiled all of these into about a hundred different text prompts and assigned each to a corresponding report type as its "intent". This array of prompts and intents became a rudimentary training data set for the model.

For this NN, I set up three nodes in TensorFlow, each with 4 inputs to represent 4 different report types that could be predicted. Once again I used a linear regression loss function and the Adam optimizer to compile the model. The unique part about this model compared to my first example was that I used TensorFlow's Universal Sentence Encoder to create the embeddings for the language model. I was surprised by how easy this was to implement. I just loaded the USE and used the models `embed` method to feed it all of the test prompt data.

With the NN set up and the model trained, I could now feed the endpoint a prompt and TensorFlow would return a set of 4 float values, 0 to 1. Each number corresponded to a report type, and its value was the percent likelihood that the prompt was intended for that report. With some simple TypeScript logic, the endpoint would return a URL for the report type with its value closest to 100%.

My hope was that the team understood how TensorFlow streamlines the setup of a NN and model training for language models, and how we could apply this to real-world use-cases going forward.

To end my presentation, I wanted to help them see visually what TensorFlow was doing, and what some bigger possibilities could be. I found a really cool tool called TensorFlow's Embedding Projector. It uses a 3D visualizer to display data points and correlation in several examples of ML models. A couple examples I used for my presentation were Word2Vec All (showing correlation between words), and Mnist with images (showing how handwritten numbers correlate to actual number values).

I set up a repo for the presentation materials I used here:

https://github.com/jdillman1989/tfjs-express-demo

I recently gave a presentation and demo for my current application development team about Neural Networks. It was an opportunity to show the team in a very practical, hands-on way the basics of NN and how they might be used. I chose to use real-world data that the team regularly works with, and built a fullstack demo of what it could look like from our development perspective, and from a user perspective.

It's uncommon to see TensorFlow implementations outside of Python, but my team has more experience in TypeScript/JavaScript. I thought it would help them grasp the basic concepts seeing it within their wheelhouse. I got surprisingly good results with a simple ExpressJS and React stack.

I set up two endpoints in the ExpressJS server to demonstrate two different use-cases: a price calculator and plain-language search predictions. The price calculator is a very bare-bones TensorFlow implementation just to introduce some of the high-level concepts to the team. It's a single node that takes a 1-dimensional input, with X values being regular pricing of a shipper, and Y values being historic sale allowances on that shipper. I used the Adam optimizer in TensorFlow to compile the model because of the relatively few training iterations I was doing for the presentation. It uses a simple linear regression loss function to predict what the future sale allowance will be for a given shipper based on the annual ad campaign it is part of.

I used data from two different annual ad campaigns, one with high correlation and one low correlation, to demonstrate how different loss values can affect the resulting model. My hope was that this would show the team a very practical example of some TensorFlow basics.

The second NN I created was to demonstrate the basics of a language model. I will go over that one in a following blog post, including some other helpful materials I used in my TensorFlow presentation.

I set up a repo for the presentation materials I used here:

https://github.com/jdillman1989/tfjs-express-demo

As a for-fun side project, I've been working on a simple top-down pixel art game engine in vanilla TypeScript on-and-off over the past year. It's been a nice break to work on TypeScript code that is completely decoupled from any framework or library, and I've always been interested in game development. Some features I want to highlight in the game engine:

1. Function- rather than Class-based code

   I see this as the trajectory that coding is moving towards, especially in web development. It takes some getting used to, and I could do a bit more structuring of the code. But I like how accessible all of my logic is.

2. Fully animated using HTML5 Canvas and requestAnimationFrame

   I learned that I needed to couple game object states with realtime Date methods rather than frame renders. A variety of factors can affect the browser window's framerate, like the device's battery level and GPU usage. I don't want the game to feel like Neo's bullet-time in those conditions. So objects can change their animations in real time, but the actual frame of the animation that's drawn is tied to framerate through `requestAnimationFrame`.

3. Color values that can be manipulated based on material and lighting data

   All the visuals of the game are stored as arrays of RGB values, rather than something like PNGs. Because of this I have full and immediate access to manipulate any color values of any visual element. If a character equips a helmet, or upgrades their weapon, I can modulate RGB values to display that change on the character. If a character is in a dark area and moves under a light source, I can change how shadows look on the character, and even adjust the colors based on the color of the light source. This lets the game display an enormous variety of visual states without having to manage a huge number of assets.

4. Visual effects for wind and weather

   I can generate sine waves that help calculate the position and animation of certain objects like cloaks and capes. It's an easy and convincing way to animate something blowing in the wind. Wind values can adjust the amplitude, frequency, and radius of the sine waves. Wind values change based on randomly calculated large circles that move across the map. Objects within those circles will change their animation frames or sine wave parameters based on the strength of the circle's wind value. There is also a day/night cycle that adjusts global ambient light values. Global ambient light values can also trigger events at certain thresholds, like lanterns turning on or fireflys lighting up.

5. Reusable menu systems for settings and inventory

   I can calculate how big I should draw a menu window based on the contents it will have. Menu and inventory options can be set up in various grids that the player can navigate using a keyboard or controller. I use the same code to render text for objects like interactive sign posts.

6. Controller support

   I use the native browser gamepadconnected event, and the navigator.getGamepads method to access connected controllers. I can read the input state of a controller and map those inputs to various functions within the game.

The basic architecture of the project is vanilla TypeScript for the front-end game engine, ExpressJS backend for basic CRUD operations on game data, and a MongoDB database to store game data. I have not built anything that could be considered a real game with this yet. There's a lot that should be done still, like save data for players, drawing larger maps, and implementing web sockets. The work has been fun, and it's enjoyable to see it play out on screen. Someday I'd like to take all of this work and turn it into a real web-based game.

Here's the repo for the game engine:

https://github.com/jdillman1989/garden-quest

Here's an example of what the game can do currently:

https://garden-quest.jdillman.com

If you haven’t read parts I and II of my JAMStack posts yet, make sure you read those first!

I want to dive into some of the more significant benefits I’ve seen working in JAMStack. I’ll be going over performance, low overhead costs, and security.

My current full time work focuses mainly on maximizing SEO for a large number of monetized websites. The majority of the work I’ve done so far in JAMStack has been converting these sites from WordPress over to JAMStack for performance gains (and subsequently gains in Google ranking). JAMStack is all about cutting the fat from a website. It let’s you completely cut out the major performance bottlenecks, like databases and backend logic, that plague other stacks. This also helps reduce the time I spend testing performance, as I can be more focused on front end performance tests. Every site I’ve converted so far has seen performance increases that would be impossible with any other stack. This report is an example of one site’s before and after:

Every metric of site performance improved substantially after conversion. Most importantly, it went from an unacceptable 12 second load time (read more about the 10 second threshold for websites) to a blazing 2.1 seconds. For this site, that directly correlates to higher search rankings and more conversions.

I briefly went into hosting costs for JAMStack sites in my last post. Because JAMStack sites ultimately compile down into static HTML files, they are incredibly easy to host compared to sites that need a backend language and databases. Using this blog site as an example, I have a free Algolia account that handles the search functionality, a free Webtask account that handles submissions to my freelance work form, and a free Forestry account that let’s me easily write content for this site. I could also easily host this site on Netlify for free (but I like having my own dedicated server for other projects I work on). The only things I have to pay for here are the jdillman.com domain name and email. The reason all of these services are free or very low cost is because JAMStack is all about each part of a site being as focused as possible. You’re never using resources or bandwidth you don’t need.

One last benefit to JAMStack that should be very apparent is its security. I’ve seen WordPress sites hacked or compromised hundreds of times, even when they are built by extremely diligent developers. And the target for hacks on these sites in every single instance I’ve seen is the database. So because there is no database in JAMStack, there’s almost nothing left to hack. There’s no SQL injection, very few access points to hosting vulnerabilities, and no CMS maintenance to always keep up with. In order to compromise a JAMStack site, a hacker would have to be able access something like GitHub or a server shell. I’m not saying it’s completely impossible for this to happen, but it’s undeniably safer than a typical WordPress setup.

And with that, I am wrapping up this series of posts on my experience with JAMStack. My bold prediction is JAMStack technology will outpace alternatives like WordPress for custom web development in the next decade. At the very least, I believe all web developers should be keeping their eyes on JAMStack and experimenting with its capabilities.