Posts in JAVASCRIPT

I recently gave a presentation and demo for my current application development team about Neural Networks. It was an opportunity to show the team in a very practical, hands-on way the basics of NN and how they might be used. I chose to use real-world data that the team regularly works with, and built a fullstack demo of what it could look like from our development perspective, and from a user perspective.

It's uncommon to see TensorFlow implementations outside of Python, but my team has more experience in TypeScript/JavaScript. I thought it would help them grasp the basic concepts seeing it within their wheelhouse. I got surprisingly good results with a simple ExpressJS and React stack.

I set up two endpoints in the ExpressJS server to demonstrate two different use-cases: a price calculator and plain-language search predictions. The price calculator is a very bare-bones TensorFlow implementation just to introduce some of the high-level concepts to the team. It's a single node that takes a 1-dimensional input, with X values being regular pricing of a shipper, and Y values being historic sale allowances on that shipper. I used the Adam optimizer in TensorFlow to compile the model because of the relatively few training iterations I was doing for the presentation. It uses a simple linear regression loss function to predict what the future sale allowance will be for a given shipper based on the annual ad campaign it is part of.

I used data from two different annual ad campaigns, one with high correlation and one low correlation, to demonstrate how different loss values can affect the resulting model. My hope was that this would show the team a very practical example of some TensorFlow basics.

The second NN I created was to demonstrate the basics of a language model. I will go over that one in a following blog post, including some other helpful materials I used in my TensorFlow presentation.

I set up a repo for the presentation materials I used here:

https://github.com/jdillman1989/tfjs-express-demo

As a for-fun side project, I've been working on a simple top-down pixel art game engine in vanilla TypeScript on-and-off over the past year. It's been a nice break to work on TypeScript code that is completely decoupled from any framework or library, and I've always been interested in game development. Some features I want to highlight in the game engine:

1. Function- rather than Class-based code

   I see this as the trajectory that coding is moving towards, especially in web development. It takes some getting used to, and I could do a bit more structuring of the code. But I like how accessible all of my logic is.

2. Fully animated using HTML5 Canvas and requestAnimationFrame

   I learned that I needed to couple game object states with realtime Date methods rather than frame renders. A variety of factors can affect the browser window's framerate, like the device's battery level and GPU usage. I don't want the game to feel like Neo's bullet-time in those conditions. So objects can change their animations in real time, but the actual frame of the animation that's drawn is tied to framerate through `requestAnimationFrame`.

3. Color values that can be manipulated based on material and lighting data

   All the visuals of the game are stored as arrays of RGB values, rather than something like PNGs. Because of this I have full and immediate access to manipulate any color values of any visual element. If a character equips a helmet, or upgrades their weapon, I can modulate RGB values to display that change on the character. If a character is in a dark area and moves under a light source, I can change how shadows look on the character, and even adjust the colors based on the color of the light source. This lets the game display an enormous variety of visual states without having to manage a huge number of assets.

4. Visual effects for wind and weather

   I can generate sine waves that help calculate the position and animation of certain objects like cloaks and capes. It's an easy and convincing way to animate something blowing in the wind. Wind values can adjust the amplitude, frequency, and radius of the sine waves. Wind values change based on randomly calculated large circles that move across the map. Objects within those circles will change their animation frames or sine wave parameters based on the strength of the circle's wind value. There is also a day/night cycle that adjusts global ambient light values. Global ambient light values can also trigger events at certain thresholds, like lanterns turning on or fireflys lighting up.

5. Reusable menu systems for settings and inventory

   I can calculate how big I should draw a menu window based on the contents it will have. Menu and inventory options can be set up in various grids that the player can navigate using a keyboard or controller. I use the same code to render text for objects like interactive sign posts.

6. Controller support

   I use the native browser gamepadconnected event, and the navigator.getGamepads method to access connected controllers. I can read the input state of a controller and map those inputs to various functions within the game.

The basic architecture of the project is vanilla TypeScript for the front-end game engine, ExpressJS backend for basic CRUD operations on game data, and a MongoDB database to store game data. I have not built anything that could be considered a real game with this yet. There's a lot that should be done still, like save data for players, drawing larger maps, and implementing web sockets. The work has been fun, and it's enjoyable to see it play out on screen. Someday I'd like to take all of this work and turn it into a real web-based game.

Here's the repo for the game engine:

https://github.com/jdillman1989/garden-quest

Here's an example of what the game can do currently:

https://garden-quest.jdillman.com

If you haven’t read parts I and II of my JAMStack posts yet, make sure you read those first!

I want to dive into some of the more significant benefits I’ve seen working in JAMStack. I’ll be going over performance, low overhead costs, and security.

My current full time work focuses mainly on maximizing SEO for a large number of monetized websites. The majority of the work I’ve done so far in JAMStack has been converting these sites from WordPress over to JAMStack for performance gains (and subsequently gains in Google ranking). JAMStack is all about cutting the fat from a website. It let’s you completely cut out the major performance bottlenecks, like databases and backend logic, that plague other stacks. This also helps reduce the time I spend testing performance, as I can be more focused on front end performance tests. Every site I’ve converted so far has seen performance increases that would be impossible with any other stack. This report is an example of one site’s before and after:

Every metric of site performance improved substantially after conversion. Most importantly, it went from an unacceptable 12 second load time (read more about the 10 second threshold for websites) to a blazing 2.1 seconds. For this site, that directly correlates to higher search rankings and more conversions.

I briefly went into hosting costs for JAMStack sites in my last post. Because JAMStack sites ultimately compile down into static HTML files, they are incredibly easy to host compared to sites that need a backend language and databases. Using this blog site as an example, I have a free Algolia account that handles the search functionality, a free Webtask account that handles submissions to my freelance work form, and a free Forestry account that let’s me easily write content for this site. I could also easily host this site on Netlify for free (but I like having my own dedicated server for other projects I work on). The only things I have to pay for here are the jdillman.com domain name and email. The reason all of these services are free or very low cost is because JAMStack is all about each part of a site being as focused as possible. You’re never using resources or bandwidth you don’t need.

One last benefit to JAMStack that should be very apparent is its security. I’ve seen WordPress sites hacked or compromised hundreds of times, even when they are built by extremely diligent developers. And the target for hacks on these sites in every single instance I’ve seen is the database. So because there is no database in JAMStack, there’s almost nothing left to hack. There’s no SQL injection, very few access points to hosting vulnerabilities, and no CMS maintenance to always keep up with. In order to compromise a JAMStack site, a hacker would have to be able access something like GitHub or a server shell. I’m not saying it’s completely impossible for this to happen, but it’s undeniably safer than a typical WordPress setup.

And with that, I am wrapping up this series of posts on my experience with JAMStack. My bold prediction is JAMStack technology will outpace alternatives like WordPress for custom web development in the next decade. At the very least, I believe all web developers should be keeping their eyes on JAMStack and experimenting with its capabilities.

So in my last post (Part I below), I went over my experience researching and building JAMStack sites with Jekyll, Algolia, and Webtask. In this second part of my JAMStack posts, I want to go over what I do with a JAMStack project after I have it built and ready to deliver. I’ll be going over the services I use for hosting, deployments, and content management.

One of the biggest benefits of JAMStack sites is that, as static sites, they are incredibly simple and easy to host. There’s no database to run, no backend language to install or configure, and no logic that needs to run every time the site is accessed. It’s just plain HTML files to be delivered. Because of these benefits and the growing popularity of static sites, there are services popping up that are dedicated to hosting static sites at a fraction of the cost of traditional hosting. The one I use for all my JAMStack projects is Netlify. My personal Netlify account that I use to host a small number of sites is absolutely free, because of how little it costs them to host these sites. At a more enterprise level, it costs the same to host hundreds of static sites on Netlify as it does to host a single WordPress site on a traditional host in my experience. But for me, the savings aren’t even the best part of using Netlify.

Netlify offers a wide array of services specific to static sites in addition to hosting, like deployment previews, form submission handling, and A/B testing. In my case, I connect a Jekyll site’s GitHub repo to my Netlify account. Netlify will take that code, build out the static site from it, and deploy that static site to a server for me. The really cool part is that Netlify will run that deploy process for every git branch I have in that repo, and assign each branch a separate placeholder url for me to access. So I am automatically able to test new features that I have branched off the main site before I merge them back in. There’s no provisioning a separate staging environment to test and QA code, I create the branch and all of that is done for me. I cannot even describe how beneficial this feature has been for me coming from WordPress development. And that’s not all, Netlify can even use these separate branch deployments to create A/B tests.

So I’ve covered how JAMStack can be just as powerful as WordPress for building templates and functionality, but what about content? WordPress’ excellent admin interface is its main selling point for many people.

Because there’s no backend functionality on a JAMStack site, it will not have a CMS built in like WordPress. We again need to call on an external service to function as our CMS. Having tried several of these headless content management systems, the most WordPress-like one I’ve found is Forestry. Similar to Netlify, I can connect my GitHub repo of a JAMStack project to my Forestry account (also free). Forestry will then read the repo and create a model of the content in a user interface very similar to the WordPress admin. As content is created and changed through the Forestry interface, it will make git commits directly back to the repo. So the site’s content is versioned through an actual VCS, which I’ve found to be very helpful compared to WordPress’ revisions functionality.

Another very helpful feature unique to Forestry is Front Matter Templates (FMTs). I mentioned in my last post how front matter is Jekyll’s equivalent to post meta for storing data like titles, dates, templates, and any other custom values you add. Using FMTs, Forestry will create an intuitive user interface for controlling the values of each different front matter. So I can create a Forestry config file in the site code which specifies that a particular front matter should display as a checkmark in Forestry, or a dropdown list of options, a calendar date, a sortable list, a file upload, or many other options. This is where a static site can really become fully fleshed out with complex layouts and still have easy-to-manage content. Similar to Advanced Custom Fields in WordPress, this gives me a lot of control over how I want to control content editing.

Admittedly, Forestry’s interface isn’t always as straightforward or helpful as the WordPress admin, but it has already come a long way in the short amount of time I’ve been using it. And I know personally that there is a very dedicated and talented team behind its development.

With these last two posts, I covered pretty much everything at a high level that I wanted to say about how I work in JAMStack. In my next (and hopefully last) post on this topic, I want to dig deeper into the benefits of JAMStack, and go over what all this means for me as a developer.

The vast majority of my career in web development as revolved around WordPress. There’s been a relatively recent paradigm shift in web development called the JAMStack that more and more developers are embracing, including myself. I’ve been pretty deep in the JAMStack world now for close to a year, and so I want to share my thoughts and experience transitioning from WordPress. I have a lot on my mind regarding this topic, so I’m planning for this to be a series of upcoming posts. In this first post, I want to give a brief overview of what JAMStack is, and some of the tools I’ve used with it.

Unlike many other tech stacks, JAMStack doesn’t really define specific technologies to use. It’s more like a theory on how to maximize performance and organize the components of a website. JAM stands for JavaScript, APIs, and Markup. These three general components are used with a static site generator to create websites that can be just as functional as any WordPress site, without all the extra bulk.

There are many static site generators that can be used in a JAMStack build, like Hugo, Gatsby, and Jekyll. Jekyll is probably the most widely used of these, and the one I have the most experience with. With Jekyll, I build out HTML templates for a site using Liquid templating, and add the site’s content to simple plain text, html, or markdown files. Once a build is triggered for the site, Jekyll’s Ruby framework takes that content and applies it to the appropriate templates, creating a directory of completely static content that’s ready to serve.

With Liquid templating and simple Ruby extensions, Jekyll is capable of everything a WordPress theme can do. Jekyll’s collections function the same as WordPress’ custom post types, front matter functions the same as custom meta, Liquid blocks work like shortcodes, Ruby plugins work like filters, and yaml configs can function the same as PHP globals. Jekyll makes the markdown part of JAMStack extremely powerful, but its lack of an actively running backend means it can’t do some of the things that WordPress has built-in like search and form submissions. This is where the JavaScript and API parts of the JAMStack come in.

One of the major selling points of JAMStack for me was how well-organized it requires code. JAMStack requires a very clear separation of frontend assets and backend functionality. In most applications of JAMStack, the frontend and backend of a site rarely even exist on the same server. JAMStack relies on the fact that there are countless API services a site can call on to provide almost any kind of backend functionality it needs. One such service that I’ve used pretty extensively is Algolia. I use Algolia to create an external index of a site’s content, and sometimes other external data that is used in the site. I can use Algolia’s index as a kind of database for the site, which lets me easily implement search functionality to a site without having to write my own algorithms. This very blog uses an Algolia index for the search bar in the menu. In addition to a site’s own static content, I’ve also used it to index data in Google Sheets to be applied across many different JAMStack projects. From the Jekyll side, this all boils down to a basic AJAX http request, and parsing through the JSON Algolia gives back.

There’s an API service out there for just about anything you’d want your website to do. But relying on these external APIs can sacrifice some of your ownership of a site’s functionality. What if a project requires a highly custom way to handle form submissions, or needs a highly custom authentication process? This is where serverless functions come in.

Serverless functions is an even newer part of web development than static site generators. It’s a way to build out backend code in a static site without having a server for the code to run on. “Serverless” is kind of a misnomer because ultimately there is a server that the code runs on, but it removes all of the maintenance, scaling, security, and other sysadmin aspects of running a server. Put way too simply, it’s a service you use to host your own API code. The biggest example of one of these services is Amazon Lambda, but so far my need for serverless functions has been too sparse to justify that particular service. I’ve been using a similar but simpler service called Webtask. As an example of how I’ve used Webtask, one particular JAMStack project I worked on needed its own custom form submission and admin view to display those submissions. I wrote the form submission process and admin authentication process in Node and Express, and saved that code into a directory in the Jekyll project that was configured to not build out with the rest of the site templates and content. I then took the config file that Jekyll uses to build the static site, and added a property to the build process that pushes the Node directory up to my Webtask account. From there, anytime a site visitor accesses the form, the form will make a basic request to Webtask, which will then handle scaling up a server on the fly and running my code. It’s incredibly powerful, and incredibly liberating as a developer.

Hopefully it’s clear from this post how powerful JAMStack can be when applied with the right tools. There’s still a lot more I want to go over about JAMStack, like hosting, deployment, and content management systems, but this is already my longest post ever. I think those topics are better left for another post. I will be working on having a Part II in the coming days.