Basic Assignments
 
Options & Settings
 
Main Time Information
Color Code: Yellow
Assigned To: Brandon Moore
Created By: Brandon Moore
Created Date/Time: 12/3/2020 11:13 am
 
Action Status: Blank (new)
Show On The Web: Yes - (public)
Priority: 0
 
Time Id: 7198
Template/Type: Brandon Time
Title/Caption: Meeting with Wayne about servers
Start Date/Time: 12/8/2020 10:00 am
End Date/Time: 12/8/2020 11:30 am
Main Status: Active

Sorry, no photos available for this element of time.


Notes:

There were 5 of us on the meeting. We had Wayne, Alan, Steve, John M, and myself. Wayne started out and we moved through a bunch of topics dealing with serves and where we are headed and what is needed. Great meeting. Here are some of my notes.

- Most of the servers have been moved over to Hostek.

- The servers are being more stable.

- We are still having problems with 3rd party solutions.

- Our internal API socket connections have slowed down - we ended up pushing some of that back on the outside clients.

- We have moved the WordPress stuff on to its own server.

- Data 0 is working much better and being more stable.

- Wayne and the guys (John and Danny) are working on some database projects.

- We are getting close to being able to cluster those servers. Currently, we are somewhat of a 1-1 server/database setup. We would like to get into clustering of those servers.

- We are getting good read and good data back on servers and database usage using Fusion Reactor.

- The bigger clients will get what they want and our smaller servers will be able to be consolidated.

- Data 2 (new data 20) seems really busy on the database queries and requests. There are also some other servers that get hit really hard.

- From Wayne, reward points seem to be a bigger lift. We have to look up so many records.

- Cloud Flare has a hard 100 second page load limit. We still have a few pages that are big pigs. We are looking into some long running report handoffs and finishing the process behind the scenes and then either emailing it or making a new link available. Dealing with Cloud Flare, we have been happy with a ton of bad traffic that is being blocked.

- Maybe getting into threads (Alan and Wayne looking into things)

- Wayne was showing us some server stats. Lots of traffic and requests.

- We have been renaming servers and changing their domain names.

- Stats on servers, requests, browsers, pages, operating systems, device types, etc. Great stats.

- Wayne was even reporting on hack attempts and measures that we are taking to either monitor and/or eliminate those attempts. It is great to have Wayne on the servers, watching and helping to smooth things out.

- Talks about databases and having unique datasources on the same servers. This is a future project.

- Talking about internal coding practices, testing, and stomping out bugs.

- Wayne, John, and Danny are working on some big database tables and consolidating those queries and data. Along those lines, they, that team, is going to do some automated testing on the new changes. The goal is to eventually get lots of automated testing in place and ready to go. Wayne was talking about processes that were looking at automated tests, and auto deployment options, if things pass all the tests.

- Introducing bugs by making a change and then that bugs shows up on some other page (what pages affect other pages).

- Talking about bugs... how do we track those, who fixes those, and how we get those bugs assigned to the correct developers? Currently, lots of the bugs are being passed over to Wayne and that's not really fair. What are our options? We need to pick somebody to start and track that process. We are thinking Cory, right now.

- Maybe using bug tracking stuff that already exists on bit bucket (Jira) or we could write our own bug tracking pieces. There are pros and cons to both scenarios.

- We may need a full-time bug person. We have ways of seeing some of those bugs even before they get really crazy. The biggest problem is who had the time and who's job is it to address those issues? That is a missing piece right now. Some of these bugs relate to our development process and lots of copying and pasting. We are seeing many of the errors show up in logs and such, but no one is actively watching or trying to take care of that.

- We tend to use junior devs (developers) and send them into a page, have them make some changes, and then go from there. It would be awesome to help them know and learn the why.

- We would like to make a standard operating procedure (SOP) for how to deal with bugs. Also, it could be training, operator errors, settings, processes, procedures, etc. We will start getting into this. We had some discussions on some of internal issues that we have. Who will fix it, what data was passed in, how can we duplicate those errors, etc.

- Two possible roles - one who collect and logs the errors. The other is a developer who can get in and get things fixed. Along with bugs, we need to also set priorities for those bugs and those bug fixes. We may need more than just one bug guys. We may need to pick and choose based on the bugs.

- Getting Wayne a mini me helper - how to jump on the servers, interact with Hostek, rebooting servers, resetting services, stopping the auto deploys, etc. We will be starting out with John Maestas as a mini me helper for Wayne.

- Data normalization rules and redoing some of the database tables. On this first round, we will do the minimum levels to get it ready to do clustering. We will normalize where possible.

- On the server side, the database are the weakest link. We would like to get these clustered and backed up even better than they currently are. Wayne and John may be looking into this as we go forward. Currently, both Hostek and Newtek are doing daily back-ups of the data, but sometimes a lot of data passes in just one day. We may need to get a smaller increment for our back-ups.

- Light talks on server caching and calls to the database. Trying to be more efficient. Limiting the number of requests per page.