Investigate a Redis cluster
Note This article contents haven’t been verified by running redis nodes. Please read official documents to verify the correctness of this document. tl; dr This is about a gist of this article.
January 19, 2025
React reference
The quick reference for a React Hooks A react hook must be called at the top level of React’s components. useState useRef useRef is a useful for a value that’s not needed for rendering, because changing a ref does not trigger a re-render. This is particularly used to handle DOM elements in React. See the officla document for more details.
June 22, 2024
Getting Started Stable Diffusion with LoRA models
1. About Stable Diffusion There are various terminologies and explanations about stable diffusion and LoRA, so it’s better to read them first: I’m also a beginner about these topics. Hugging Face: Using LoRA for Efficient Stable Diffusion Fine-Tuning Anakin.ai: How to Use Lora in Stable Diffusion - A Step-by-Step Guide Anakin.ai: How to Use Stable Diffusion Checkpoints - A Complete Guide In this article, I followed the article in Anakin.ai mostly.
May 24, 2024
Cheat sheet for programming languages
Basics PHPPythonGolangTypeScript null None nil null Undefined key undefined Equal $var1 == $var2 var1 == var2 var1 == var2 var1 == var2 Strict equal $var1 === $var2 var1 === var2 Null Coalescence $var ?? 'default' var ?? 'default' Null Assertion (Ignore null value) var! Strict equal in TypeScript 2 variables have the same types Syntax for types PHPPythonGolangTypeScript Type Alias type NewType OldType type VariableType = OldType type FunctionType = (var: number) => void Interface interface Interface { method(var int) void } interface VariableInterface { var: number; } interface FunctionInterface { (var: number): void; } The difference of type and interface between TypeScript:
April 28, 2024
Linux command cheetsheet
Basic commandsgrep grep’s regular expression is basic regular expression as default. -$NUM, -C, --context $NUM, -B, --before-context $NUM, -A, --after-context $NUM: show the $NUM number of lines around/before/after the selected line -v, --invert-match $REGEXP option: Select non-matching lines. -E, --extended-regexp $REGEXP option: Use extended regular expressions (EREs). See this document for more details of the differences between basic and extended regular expressions. -P, --perl-regexp $REGEXP option: Use Perl-compatible regular expressions (PCREs). sed -n, --quiet, --silent: Stop printing inputs -i, --in-place $SUFFIX: Edit files in place -E, -r, --regexp-extended: Use extended regular expression. sed expressions Commands s/$BEFORE/$AFTER/, s/$BEFORE/$AFTER/g: replace a pattern $BEFORE with $AFTER. The separate / can be replaced with anything as long as the same separator is used, like s%/% / %. Adding g replaces all patterns matching instead of only the first pattern. p: Print selected address ranges. Use like ${LINE}p, or /pattern1/,/pattern2/p Address ranges $LINE_NUM $LINE_NUM_FROM,$LINE_NUM_TO: Select lines from $LINE_NUM_FROM to $LINE_NUM_TO awk -F is field separator. The default value is a 0x20, which matches a space, tab, and newlines this article. There can be some generic syntaxes like Variable can be defined and used Internal variables: { var=$1; print var } Conditional flow like if (condition1) { statement1; } next can skip a line xargs envsubst Use casesHow to output a line before other lines. Using awk,
March 30, 2024
Getting Started with Golang text/template
Template packageFirst example It outputs the result to Stdout Insert variables into a template Use Go playground {% raw %} type TemplateVariables struct { Name string } tmpl := template.Must(template.New("template").Parse(`{{ .Name }}`)) err := tmpl.Execute(os.Stdout, TemplateVariables{ Name: "name inserted", }) if err != nil { return err } return nil {% endraw %}
August 5, 2023
Follow Kubernetes the Hard way
This article was written by just following Kelsey Hightower’s Kubernetes Hardway document to understand Kubernetes internal architecture. OverviewControl plane The control plane’s overview: Each node on the control plane: etcd: Store cluster states 2379: Client API port 2380: Peer API port Kubernetes API Server 6443 Scheduler Controller Manager nginx For HTTP heath check from GCP Load Balancer Each middleware was installed by systemd. Data plane
June 19, 2023
GCP Billing Analyze
There are a few documents to manage billing data in BigQuery Attribution of committed use discount fees and credits How to export to BigQuery Structure of the standard data QueriesSum of all costs From the official document: SELECT invoice.month, SUM(cost) + SUM(IFNULL((SELECT SUM(c.amount) FROM UNNEST(credits) c), 0)) AS total, (SUM(CAST(cost AS NUMERIC)) + SUM(IFNULL((SELECT SUM(CAST(c.amount AS NUMERIC)) FROM UNNEST(credits) AS c), 0))) AS total_exact FROM `project.dataset.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX` GROUP BY 1 ORDER BY 1 ASC ; How to calculate commitment fees From the official document:
May 3, 2023
Getting started cgroups for kubernetes resource requests and limits
OverviewThere are many excellent articles or videos describing what is Kubernetes CPU resources requests and limits and how they are implemented. Watch followings to understand how the CPU resource requests and limits of a container on the Pod spec works on a Linux host with cgroup. Kaslin and Kohei’s video in CNCF in 2021 A YouTube video The slides of this video Shon’s articles CPU resource requests CPU resource limits {% comment %} https://blog.kintone.io/entry/2022/03/08/170206 https://medium.com/omio-engineering/cpu-limits-and-aggressive-throttling-in-kubernetes-c5b20bd8a718 https://engineering.squarespace.com/blog/2017/understanding-linux-container-scheduling {% endcomment %}
April 27, 2023
Manage kubernetes StatefulSet
OperationsIncrease the storage sizes without downtime We cannot update the resources.storageSize of the StatefulSet to increase the volume sizes. So, we have to follow a few steps to update them. See this article for more details. Confirm if the storageclass of persistent volumes has .allowVolumeExpansion=true > kubectl get storageclass standard -o yaml | yq .allowVolumeExpansion true ``` Update all PVC to have more storage sizes
March 23, 2023
Getting Started OpenTelemetry on Golang
Getting StartedAt first, develop a tutorial in the official document and develop a go clients.
March 19, 2023
The Basics of Elasticsearch
Following videos are helpful to understand the overview of Elasticsearch more. Elasticsearch architecture About search relevance Documents and index Document: JSON object, equivalent to a row in a table of RDBMS Index: The set of documents collected by the same type of data. For exaple, one index is for a user, second one is for a product in an e-commerce service. Elasticsearch Cluster
March 15, 2023
Getting Started with Grafana Tempo
Grafana TempoFollow this example. Install grafana tempo with microservices-tempo-values.yaml. I commented out storage values because when I tried to use GCS, it failed due to the permission issue to access GCS Added the grafana tempo as the data source of Grafana and see if it works The service is -gateway created by the helm chart Update Grafana OperatorI deployed the Grafana by grafana-operator on this post. So, it was required to update Grafana by following this example
March 12, 2023
Prometheus Metrics Overview on Grafana
In this post, some variables defined in Grafana are used for Prometheus metrics, including $__rate_interval: This article describes the benefit of this variable Kubernetes MetricsThese metrics require installing some of followings: Node Exporter kube-state-metrics cAdvisor Node metrics CPU utilization per node: 1 - (avg by (instance)(rate(node_cpu_seconds_total{mode="idle"}[$__rate_interval]))) Memory utilization per node: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) Disk utilization per node: 1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes) Number of pods with a certain phases on a node (from this comment):
March 12, 2023
Set up a Grafana operator
Follow this document mainly. Set up a GrafanaFirst, clone a repository and apply a change. kubectl kustomize deploy/manifests -o grafana-operator.yml kubectl apply -f grafana-operator.yml Then create a Grafana resource in the same namespace. I tried to create it in the different namespace, but didn’t work for some reasons.
February 27, 2023
Use Google Secret Manager in a GKE cluster
There are an awesome article about the options to use the Google Secret Manager and their pros and cons. In this article, use Secrets Store CSI Driver by following this page. Set up a secrets store CSI driverFirst, install the secrets-store-csi-driver in kube-system namespace helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system Next, install the GCP provider for the secrets store CSI driver from this repository. Unfortunately, there is no helm chart as the time of this post is written, according to this GitHub issue.
February 26, 2023
Set up a config sync in a GKE cluster
Basically, follow this document Enable a Config Sync Select **Install Config Sync > your cluster" Clear the Enable Policy Controller checkbox and click Next. Leave the Enable Config sync checkbox enabled. In the Repository list, select “Custom”. Set your repository on your URL and click “SHOW ADVANCED SETTINGS” Choose how to read a GitHub repository by one of the ways described in this document. I used a GitHub Token which will be expired shortly. And set it out on our cluster. Set the Configuration directory as the root directory on the repository. I set config-sync Change the Source format to hierarchy. You can see the details about it in here. Set up a GitHub repositoryCreate files under the config sync directory set above and see if the config sync works. The guestbook is the manifest set no the other getting started in this post.
February 25, 2023
Set up a prometheus by a prometheus operator in a kubernetes cluster
This document is written by following this document. Install a prometheus operatorFirst, create CRDs and resources in the default namespace. set LATEST (curl -s https://api.github.com/repos/prometheus-operator/prometheus-operator/releases/latest | jq -cr .tag_name) curl -sLO https://github.com/prometheus-operator/prometheus-operator/releases/download/$LATEST/bundle.yaml kubectl apply -f bundle.yaml Note that I failed to create a prometheus resource because the CRD definition was too big like next.
February 25, 2023
Set up a vertical pod autoscaler in a GKE cluster
To figure out which kubernetes objects are how much resource, a vertical pod autoscaler might be useful. It has a feature to either automatically update values, or suggest values. But it cannot be used with a horizontal pod autoscaler to update values if the horizontal pod autoscaler doesn’t use a custom or external metrics. In that case, use a multidimensional pod autoscaling. In this post, it’s to test a vertical pod autoscaler by following this document and this article.
February 25, 2023
Manage TiKV regions on TiDB
TiDB data is split into multiple nodes and they’re called the name as a region. In this document, I just played around to confirm the places and behaviors for regions on TiDB. Manage TiKV regions through PD metadata To handle PD metadata from CLI, follow this document. At first, a port-forward to access PD pods kubectl port-forward -n tidb-cluster svc/basic-pd 2379:2379 &>/tmp/portforward-pd.log & Then run following sub commands to check each. These commands are one of pd-ctl, listed in this document, and to execute them, it has to add a command like tiup ctl:v6.5.0 pd -u http://127.0.0.1:2379 store:
February 11, 2023
Online Schema Change on TiDB
There is a good video to describe a algorithms of TiDB: Also, there are a few documents related to online DDL The Online DDL design is descrbed in this design document Google F1 algorithm, which is refered from TiDB online schema change, is described in this research paper Online Schema changeIt seems it’s unable to confirm the schema version on TiKV node. Inside a PD node, it’s stored in the /tidb/ddl/global_schema_version in the etcd according to this article.
February 11, 2023
Getting Started with cue
Install go install cuelang.org/go/cmd/cue@latest Generate a YAML file from cue Prepare a sample file user.cue by a cue import "time" #User: { id: int name: string created: time.Format("2006-01-02") } john: #User john: { id: 1 name: "John" created: "2023-02-09" } Then run a command to output a yaml file by the export subcommand.
February 9, 2023
Getting Started with Kubernetes API by Kubebuilder
This document just follows a quick tutorial for kubebuilder and learn its behavior. InstallInstall kubebuilder. curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH) chmod +x kubebuilder && mv kubebuilder /usr/local/bin/ Workaround a first projectAt first, create a project and a Guestbook API, and make manifests
February 9, 2023
Working around MySQL lock metadata
There are multiple documents about innodb locks on MySQL 5.7: InnoDB locking Locks Set by Different SQL Statements in InnoDB Using InnoDB Transaction and Locking Information Granularity of locksThere are 4 types of granularity of locks Shared lock Exclusive lock Intention shard lock Intention exclusive lock Types of locks Record locks Gap locks Next-Key locks the value in the index for the next-key lock indicates as “supreme” pseudo-record, which is not a real value Insert Intention locks AUTO-INC locks This is a table lock, and how to lock the table depending on the configuration of innodb_autoinc_lock_mode Lock tables information_schema.innodb_trx information_schema.innodb_locks information_schema.innodb_lock_waits Use casesUpdate queries without an index Create a test table and insert a few records.
January 16, 2023
Upgrade Windows 10 to Windows 11
I used to use Windows 11, but for some reasons, the OS stopped working and I needed to clean-install it from Windows 10 from windows recovery environment. After installing and trying to install Windows 11, there are a few things I needed to do. Note that my motherboard was AMD B450. Enable UEFI boot on BIOSIn order to enable to UEFI Boot on BIOS, I needed to do Convert the disk from Master Boot Record, MBR, into GUID Partition Table, GPT, partition style. Disable Compatible Support Module, CSM on BIOS to enable UEFI boot Note that I didn’t have to do at the end, but I tried to enable Secure Boot on BIOS, but it prevented to boot Windows and needed to reset the bios configuration by taking its battery, described in this article.
January 14, 2023
Getting Started TiDB on GKE
See another post also to set up a TiDB on minikube. To get the basic concept and architecture of TiDB, there are many articles and videos on the Internet, like following: Set up a TiDB cluster by Kubernetes OperatorFollow the official document for GKE and the official document for kubernetes operator for this document. ext4 file system configuration It’s recommended to configure following options for better I/O performance for ext4 file system.
January 8, 2023
Getting Started Containerd
Overviewcontainerd is designed to built with a larger system like kubernetes. To see the overview, it’s better to check something like As the above video shows, it depends on plugins or runtime outside of its core by gRPC or the light weight gRPC (ttRPC), like runc for container runtime runc is a tool to manage containers on Linux following the OCI specification InstallFollowing the official document, install a containerd from the official binaries.
December 26, 2022
MySQL server configuration
LogsAbout slow queries slow_query_log = 1 slow_query_log_file = /var/log/mysql/slow_query.log long_query_time = 0 log_queries_not_using_indexes The field long_query_time is the time to output a query as a slow log. The meanings of other fields are obvious.
December 23, 2022
SSH client
SSH configurationThis is the mapping between config file /.ssh/config and ssh cli options. config file field cli option Description ForwardAgent -A or -a Enable to use locale machine’s keys on remote machines SSH passwordStore the key and access on remote machine without inputting a password. See stackoverflow for details. How to add password to the private key without a password Use: ssh-keygen -p -f /path/to/key. For example, ssh-keygen -p -f ~/.ssh/id_rsa
December 23, 2022
ProxySQL Getting Started
The official document: Docker image and initial configuration Getting StartedSee this docker-compose.yml and proxysql.cnf for the example of proxysql. With this configuration, you can access a proxysql by docker compose up -d mysql -h 127.0.0.1 -u radmin -P 16032-pradmin --prompt "ProxySQL Admin> " There are multiple ports
October 15, 2022
Getting Started with TiDB by Kubernetes Operator
Getting Started Use minikube by following this document Start minikube minikube start alias kubectl="minikube kubectl --" kubectl cluster-info Apply an operator Install its CRDs
May 1, 2022
Getting Started with Vitess by Kubernetes Operator
There are fewer lock contentions to worry about, replication is a lot happier, production impact of outages become smaller, backups and restores run faster, and a lot more secondary advantages can be realized Getting StartedDocker Follow this tutorial git clone git@github.com:vitessio/vitess.git cd vitess make docker_local ./docker/local/run.sh Kubernetes Operator Prerequisite Follow this tutorial.
May 1, 2022
MySQL backup and restore
In this article, explain how to backup MySQL database using Percona Xtrabackup. There are two binaries, innobackupex and xtrabackup. innobackupex is a wrapper for xtrabackup and it was going to be removed. It might have been deleted from version 8.0. Getting StartedInstallation Follow this tutorial wget https://downloads.percona.com/downloads/Percona-XtraBackup-LATEST/Percona-XtraBackup-8.0.23-16/binary/tarball/percona-xtrabackup-8.0.23-16-Linux-x86_64.glibc2.17.tar.gz tar xvf percona-xtrabackup-8.0.23-16-Linux-x86_64.glibc2.17.tar.gz mv percona-xtrabackup-8.0.23-16-Linux-x86_64.glibc2.17 /usr/lib export PATH=/usr/lib/percona-xtrabackup-8.0.23-16-Linux-x86_64.glibc2.17 /usr/lib:$PATH Run backup Full backup See this page for more information about full backup and a streaming backup examples.
March 19, 2022
tmux
Basic configurationThere are a few options to use set. set-option: an alias of a set -s: server option -g: global option setw or set-window-option: window option See StackExchange: Difference between global, server, session and window options for more details about these options and how they’re effective. Other references Stack overflow: What are the differences between set -g, set -ga and set-option -g in a .tmux.conf file?. Plugin managementTPM is a plugin to install plugins easily for tmux.
March 13, 2022
Getting Started with Next.js
Set up TypeScript Next.js with Material UI Follow this article npx create-next-app --typescript mui cd mui nodenv local 17.1.0 npm install @mui/material @mui/icons-material @emotion/react @emotion/styled Then use one of a material component. Dark mode or light mode Follow this official page.
February 6, 2022
setup a ubuntu desktop computer
I set up Ubuntu for the 1st time since several years ago. Set up basic configurationSettings Open Show Applications > Settings and set each configuration. Displays Set a resolution I ended up to set 1920x1080. At first, I set 3920x2160 but, there was a performance issue with my graphics card. I reduced the resolution to 2880x1620 (16:9), but there was a garbled issue on many applications. Enable Night light mode Open Night Light tab Enable Night Light toggle Mouse & Touchpad Change the scroll direction on the middle button of a mouse by turning on Mouse > Natural Scrolling.
January 23, 2022
Getting Started with MySQL Orchestrator
Orchestrator is a tool for MySQL HA and replication management. Getting started This is a simple set up document to set up an orchestrator. I just followed these documents at first. https://github.com/wagnerjfr/orchestrator-mysql-replication-docker https://github.com/openark/orchestrator/blob/master/docs/install.md Prerequisite In order to promote a read replica to a main DB, it looks GTID based replication needs to be enabled. Build a docker image At first, build a docker image. I couldn’t find which docker image in DockerHub can be used in a docker-compose.
November 7, 2021
Getting started to Consul
Consul is developed by Hashicorp to provide a few features like a service discovery. This document is to learn a little bit about setting up Consul to understand it as a first step, but not for production purpose. So far, this includes To set up consul agents To use a consul client to access the agent Look up a web server from the consul datacenter The first step for consul servers In the first step, I refer to next page.
November 6, 2021
MySQL Replication
This configuration is for the version 5.7 and it’s minimum configuration in the official document. There are 2 types of replication setup. Using binary log file positions Using Global Transaction Identifiers Binary logging replication There are a couple of important configurations server-id: The unique ID on each server, and must be a positive integer between 1 and 2^32-1. The default value is 0. log-bin: the file name for binary logs. This is required to enable replications using binary loggings. Binary loggings are not required on read replicas, but they can be used for data backups and crash recovery. Besides,
November 5, 2021
gh-ost
gh-ostgh-ost is an online migration tool for MySQL developed by GitHub. There is a video to describe what issue gh-ost is used to solve. Getting Started There are documents that are helpful for this article. Cheatsheet Prerequisite When gh-ost connects to a replica and migrates on master If a replication is SBR log_bin and log_slave_updates must be enabled on a read replica server binlog_format=ROW If a replication is RBR When gh-ost connects to a main DB and migrates on master A replication must be RBR Build a binary from source code > wget https://github.com/github/gh-ost/releases/download/v1.1.2/gh-ost-binary-linux-20210617134741.tar.gz --2021-07-28 15:40:14-- https://github.com/github/gh-ost/releases/download/v1.1.2/gh-ost-binary-linux-20210617134741.tar.gz Resolving github.com (github.com)... 192.30.255.112 Connecting to github.com (github.com)|192.30.255.112|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-releases.githubusercontent.com/54378638/0a93f200-cf85-11eb-8870-035762e21f3a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210728%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210728T063905Z&X-Amz-Expires=300&X-Amz-Signature=993eae9ee5e3b05bc105d62feb28ffe1d76ee2dd3c1ef7c2a77d7e16c72b3ccb&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=54378638&response-content-disposition=attachment%3B%20filename%3Dgh-ost-binary-linux-20210617134741.tar.gz&response-content-type=application%2Foctet-stream [following] --2021-07-28 15:40:14-- https://github-releases.githubusercontent.com/54378638/0a93f200-cf85-11eb-8870-035762e21f3a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210728%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210728T063905Z&X-Amz-Expires=300&X-Amz-Signature=993eae9ee5e3b05bc105d62feb28ffe1d76ee2dd3c1ef7c2a77d7e16c72b3ccb&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=54378638&response-content-disposition=attachment%3B%20filename%3Dgh-ost-binary-linux-20210617134741.tar.gz&response-content-type=application%2Foctet-stream Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.108.154, 185.199.110.154, 185.199.111.154, ... Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.108.154|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 4710185 (4.5M) [application/octet-stream] Saving to: ‘gh-ost-binary-linux-20210617134741.tar.gz’ gh-ost-binary-linux-202106171 100%[=================================================>] 4.49M 4.18MB/s in 1.1s 2021-07-28 15:40:15 (4.18 MB/s) - ‘gh-ost-binary-linux-20210617134741.tar.gz’ saved [4710185/4710185] > tar -xzvf gh-ost-binary-linux-20210617134741.tar.gz gh-ost > ./gh-ost --version 1.1.2 > mv gh-ost /path/to/bin/ Dry run a migration without a real update Without an --execute option, gh-ost runs DB migration for testing. Next CLI runs to add new column email on my_table in my_schema DB.
October 31, 2021
jq cheetsheet
jq is used to parse JSON result, format and output on the cli. This document shows brief overview for how to Basics Using variables: exp as $var | exp length as $array_length | add / $array_length Boolean operations: AND: exp and exp OR: exp or exp string functions: contains(substr) returns boolean startswith(substr) returns boolean join(separator) returns string select can be used to filter specific records from_entries can be used to parse Use casesGet the object of any key exp | to_entries | .[].value | exp Output CSV/TSV result from JSON @csv and @tsv can be used to format the array with CSV or TSV
October 31, 2021
Overviews for Apache cassandra
Apache Cassandra Column Store Structure Clusters: Container for Keyspaces Keyspace = DB in RDBMS Confiure replication strategies, for example Column Family = Table in RDBMS Column can be added at any given time Architecture explanation Read Anti-Entropy: use latest data from one node Write Append only even for updates Compaction is used not to blow disk out
July 30, 2021
IntelliJ plugin development
Getting startedSet up a project See this page for the details. Create a new repository from the template repository Update a repository to follow README Update pluginGroup, pluginId, and plugin name Update a repository for your purpose Publish a plugin manually at first It requires a review from JetBrains and need to wait for a few days Development a plugin Run a plugin in an IntelliJ IDE See an official page for more details.
July 24, 2021
Go MySQL client
MySQL connectionsSee also this article for further details. MySQL Driver configurations The repository for MySQL driver is https://github.com/go-sql-driver/mysql DSN parameters The DSN parameters for timeout is either application layer and TCP layer and they’re on client sides. timeout: a dial timeout. Actual code readTimeout: Used to set timeout for reading data from I/O. This is used for net.Conn.SetReadDeadline Usages Set ReadDeadline for a buffer Check connection for a first query from a connection pool writeTimeout: Used to set timeout for writing data from I/O. This is used for net.Conn.SetWriteDeadline Usages Code for writting a packet MySQL server configurations Even if the client connection is disconnected, queries on MySQL servers can keep running. On MySQL server side, some configurations like wait_timeout is used to close a non-interactive “idle” connection. See this answer, for example.
May 17, 2021
Setup a windows desktop computer
This is written on March 2021. I set up Windows for the 1st time since several years ago and there was a lot of difference between recent version and the version I used. This page collects information for something it would have been helpful for setting up windows for me. Setup for software developmentJoin Windows Insider Program Windows Insider Program provides users to access applications from their dev, beta, or preview releases. This is helpful to use some applications before it becomes stable releases, like WSL, and many documents on the Internet is also written for these pre-stable versions.
March 22, 2021
Notes when I updated a WSL version from 1 to 2
I mostly followed this article to update a WSL version, except that I didn’t enable Hyper-V until then and got an error Please enable the Virtual Machine Platform Windows feature and ensure virtualization is enabled in the BIOS. while I was trying to update the vesrion. These are steps I did to update the WSL version. Ran PowerShell as Administrator. Enabled the Windows Subsystem for Linux by next command. PS C:\Windows\system32> dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart Deployment Image Servicing and Management tool Version: 10.0.19041.844 Image Version: 10.0.19042.868 Enabling feature(s) [==========================100.0%==========================] The operation completed successfully. Enabled Virtual Machine Feature by next command.
March 16, 2021
Compare static site generator
There are many web sites to compare static site generator, but they miss some explanations that require to me. For some people, these features are important for static site generator, Their own domain name, i.e., we do not have to buy our own domain Markdown files can be used to publish static site pages without customizing anything GitHub private repository is also supported Price is free Optional: HTTPS is supported ComparisonFrameworks Gatsby Markdown: There is a plugin gatsby-plugin-mdx, and it’s added by npm init gatsby, but I couldn’t find how to enable it Next.js Markdown: Customization is required. See this blog There is a downside of routing a file: https://medium.com/frontend-digest/which-to-choose-in-2020-nextjs-vs-gatsby-1aa7ca279d8a Hugo Markdown: Supported and it’s included in a quick start step. price: Platform Gatsby Cloud Domain name: Maybe Price: Free Netlify Domain name: GitHub private repository Price: Reference Techrader: Best static site generators of 2021 SNIPCART: Choosing the Best Static Site Generator for 2021 SCOTCH: Top 11 Static Site Generators in 2020
March 8, 2021
PHP Benchmarks
PHPBench frameworkThere is an phpbench for php benchmarks. Set up Install: composer require phpbench/phpbench --dev Add phpbench.json under a project root directory { "bootstrap": "vendor/autoload.php" } Example of benchmark code The example to run a benchmark code. There are a few annotations for methods of phpbench. For example, - Revs: How many times code is executed within a single measurement. - Iterations: How many times measure execute the benchmark
September 3, 2020
Getting Started with Kubernetes Cronjob
ConfigurationsImportant fields .spec.jobTemplate.spec.template.spec.restartPolicy: How to handle when a container in a pod. Note that this doesn’t mean a job restarts or not (it’s commented on this github issue). onFailure: Restart a container in the same pod when it fails. Never: Restart a container with new pod. .spec.concurrencyPolicy: How to handle concurrent jobs Allow (Default): Allows concurrent runs Forbid: Does not allow concurrent runs Replace: Replace old job with new job if previous job hasn’t finished when it’s time to schedule new job .spec.successfulJobsHistoryLimit, .spec.failedJobsHistoryLimit: How many pods is kept after it succeeded or failed .spec.startingDeadlineSeconds: How long jobs keep trying to start. Also, see a pitfall section. PitfallsStartingDeadlineSeconds A cronjob stops starting jobs if it “misses” jobs more than 100 times in a specific period related with .spec.startingDeadlineSeconds. The details are explained in here.
August 16, 2020
Getting Started with Kubernetes Deployment
The deployment is many use cases and in this page, they’re not described. For the details for those use cases or the concept of deployment, see official page. kubectl commandsRevisions and rollbacks How to get revision history of deployments. To see revision history: kubectl rollout history deployment.v1.apps/[deployment name]. For example: kubectl rollout history deployment.v1.apps/test-deployment deployment.apps/test-deployment REVISION CHANGE-CAUSE 135 <none> 137 <none> 138 <none> 139 <none> 140 <none> 141 <none> 142 <none> 143 <none> 144 <none> 145 <none> 146 <none> To see the detail of a revision: kubectl rollout history deployment.v1.apps/[deployment name] --revision [revision number] > kubectl rollout history deployment.apps/test-deployment --revision 145 deployment.apps/test-deployment with revision #145 Pod Template: Labels: app=test-deployment env=prod pod-template-hash=c7d84d6fc Annotations: prometheus.io/port: 9100 prometheus.io/scrape: true Containers: test-deployment: Image: gcr.io/test-project/test-deployment:tag Port: <none> Host Port: <none> Limits: cpu: 1 memory: 256Mi Requests: cpu: 1 memory: 256Mi Environment: DEBUG: false Mounts: <none> Volumes: <none> How to rollback deployment: kubectl rollout undo deployment [deployment name] (--to-revision [revision number]). Examples: Rollback to the previous version. > kubectl rollout undo deployment test-deployment deployment.extensions/test-deployment rolled back Rollback to a specific version. > kubectl rollout undo deployment test-deployment --to-revision 2
August 16, 2020
Helm Client
Command listDeployment See revision history: helm history <release name> Rollback: helm rollback <release name> [revision] Rollback to the previous version if revision is 0. (from this github comment) TroubleshootingsHow to downgrade helm client version using Homebrew If helm versions are incompatible with servers, then helm cli can’t be used. See this issue for the details of issues and solutions. To sum up, following commands should be ran to downgrade the version to 2.11.0 In order to downgrade the helm cli version, we have to use
August 16, 2020
MySQL Performance
This document is written for MySQL 5.7, so these contents may be not correct for other versions. In this page, performance_schema is mainly discussed. Latency monitoringsThere are some tables in performance_schema to check the number of queries for the table. If you are interested in other information, please see other web pages like “What Does I/O Latencies and Bytes Mean in the Performance and sys Schemas?”. table_io_waits_summary_by_table table in performance_schema The details of this table is described in official page. This table stores all I/O wait events, including select, insert, update, and delete DMLs.
August 8, 2020
Overview about MySQL Lock
This document is written for MySQL 5.7, so these contents may be not correct for other versions. Lock timeoutWhen you run DDL or DML, in some cases, you may get next error. ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction This is related with lock_wait_timeout variable.
August 8, 2020
Git hooks
ConfigurationsTemplate directory See this article and this for details. Git template is the directory to gets copied everytime when a repository is created or cloned. Set a template directory: git config --global init.templatedir '~/.config/git/templates' Create a directory for hooks: mkdir -p ~/.config/git/templates/hooks Now we can add the hook under ~/.config/git/templates/hooks directory, like post-commit file which is executable. Pre-commitI think pre-commit hook is most useful hook of all git hooks. For example, in order to check the code is good to commit by running lint tools. Due to this, there are some useful tools to configure hooks easily.
July 24, 2020
Gitconfig
ConfigurationThe detail for gitconfig is written in official page. Conditional includes includeIf section can be used to include another git configuration under certain conditions. gitdir: If repositories are under a specific directory, glob pattern can be used. Note that the path of this should end with /. gitdir/i: The same as gitdir but case-insensitive matching, onbranch: If current branch matches the glob pattern of onbranch. For example, there are two files ~/.gitconfig and ~/.gitconfig-group.
July 24, 2020
Compare performances of gRPC server streaming and pagination of unary RPC
There are some cases that we wanna fetch all records that are matching with a certain condition from an other gRPC server. In these cases, there are at least 2 possible solutions to implement it, by gRPC server streaming or pagination. In this post, in order to check the benefit of the performance of gRPC server streaming over pagination, running benchmarks for the both of pagination and server streaming, and checked the performances.
May 17, 2020
Overview of gRPC
Written in March, 2020. gRPC is the protocol using HTTP/2. For the details, please take a look for a official page. This page explains how to use it in go and how it behaves. TypesThere are some types for RPC. unary RPC server stream RPC client stream RPC bidirectional stream RPC Using stream RPC, multiple messages can be sent on a single TCP connection. ExamplesExamples include unary RPC(SayHello) and server stream RPC(KeepReplyingHello).
May 17, 2020
bigquery usages
FunctionsString REGEXP_REPLACE Syntax: REGEXP_REPLACE(value, regexp, replacement). Returns a string value of which sub string matches with regexp is replaced with replacement. If value contains more than one substrings matching with regexp, it’s only applied for the first strings. For the syntax of regexp, we can use the syntax of re2. For more details, see official document. Use cases Remove query string from URL SELECT REGEXP_REPLACE('https://console.cloud.google.com/bigquery?project=project', '\\?.*$', ''); Date DATE_TRUNC Syntax: DATE_TRUNC(date_expression, date_part).
May 10, 2020
Enable comments on minimal mistake theme of GitHub pages
The configuration to enable comments is described in the official page. Supported providersThere are some providers to support comments, and some of them is not free. Disqus: Not free Discourse: Not free Facebook: Free Utterance: Free with GitHub issues. Staticman: Free but needs to setup our own instances like on heroku (which is explain in this issue). Utteranceutterance is easy to setup, and users require GitHub authentication to comment, and it can enable you to get notifications for issues. Furthermore, because you can manage messages on GitHub, you can manage comments on GitHub features.
April 26, 2020
gitHub pages
Getting StartedSee Official tutorial for detail steps. Basically, what you have to do is Create .github.io repository Choose theme on settings tab in GitHub Customize pagesBefore updating your repository, please see each repository here for your theme to check how to customize for your purpose. The examples of customization are Twitter share button: go to publish twitter button page. Choose good themeThere are some pages to collect the themes of GitHub pages. These are the pages I checked.
April 26, 2020
git cli
Written in March 2020. ToolsRewrite history Note that this is REALLY DANGEROUS operations and please do it with your responsibility. git-filter-repo can be used to rewrite history, and it’s official tool to replace with git-filter-branch. Install on Mac OS > brew install git-filter-repo How to use Replacing name and email with new one. Use --email-callback and --name-callback options. For more details, run git filter-repo --help. > git filter-repo --email-callback 'return email.replace(b"old@gmail.com", b"new@gmail.com")' --name-callback 'return name.replace(b"Old name", b"New name")' Parsed 174 commits New history written in 0.19 seconds; now repacking/cleaning... Repacking your repo and cleaning out old unneeded objects HEAD is now at 973ca76 Merge pull request #48 from at-ishikawa/move-grpc Enumerating objects: 1365, done. Counting objects: 100% (1365/1365), done. Delta compression using up to 16 threads Compressing objects: 100% (717/717), done. Writing objects: 100% (1365/1365), done. Total 1365 (delta 480), reused 1273 (delta 473) Completely finished after 0.67 seconds.
March 21, 2020
Getting Started with MySQL DDL
This document is described based on MySQL 5.6. Online DDLOn MYSQL 5.6, DDL can be ran online, without maintenance. There are two important types of online DDL. Algorithm COPY: Concurrent DML is not supported. New table is copied from original table and. INPLACE: Concurrent DML may be supported, but sometimes, exclusive lock is taken. Avoid copying tables to update schema. DEFAULT: Use INPLACE if it is supported for running DDL. Otherwise, COPY. LOCK NONE: Permit concurrent reads and writes, or error occurs. SHARED: Permit concurrent reads but block writes, or error occurs. EXCLUSIVE: Block reads and writes. DEFAULT: Choose one way from NONE, SHARED, or EXCLUSIVE by this priority. It’s recommended to read the official document at first to understand the more details for what happens in each operation.
March 20, 2020
Enable lunr search for non-posts on minimal mistake theme of GitHub pages
This page explains how to enable searching non-posts pages for Minimal mistakes by Lunr.js for someone who does not know jekyll at all. Lunr.js is the default search engine of the theme, so if turn on search by adding search: true in _config.yml, then you all posts become searchable by lunr.js. However, for non-post pages, you may need a different configuration, so I’ll write this page for it. The entire configuration which requires is following in this page.
March 19, 2020
Introduction to GCP Cloud endpoints
The Cloud endpoint is actually the NGINX proxy which offers the following features on GCP. Authentication and validation Logging and monitoring in GCP The overall architecture for this is described in official page. Supported protocolThey support OpenAPI. The spec of OpenAPI is described here. gRPC Rest APIs using Cloud Endpoints Framework Supported environments Endpoint on Cloud Run Endpoint on GKE as a sidecar Endpoint on App Engine For Cloud Run or GKE, docker images are available in GCP registry. You can use the image like gcr.io/endpoints-release/endpoints-runtime:1. See released and secure image versions in here.
March 19, 2020
kubectl cheetsheet
Collect recent error logsIf the logs are outputted by zap, error messages are aggregated by checking level = error. This log does not work very well if the field error contains some unique values like id. $ kubectl logs -l key=value --since=5m | jq -r 'select(.level=="error") | .error' | sort | uniq -c | sort -bgr Instead of error field, msg or errorVerbose could be used.
March 19, 2020
MySQL Tuner
MySQL Tuner tool This is a tool to review a configuration for MySQL server. Getting started To run this tool, we should run following commands. wget http://mysqltuner.pl/ -O mysqltuner.pl wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/basic_passwords.txt -O basic_passwords.txt wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/vulnerabilities.csv -O vulnerabilities.csv perl mysqltuner.pl
March 19, 2020
Nginx server configurations
Performanceproxy_cache The below configuration is the example to cache images. proxy_cache_path /path/to/cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m use_temp_path=off; server { # ... location / { proxy_cache my_cache; proxy_pass http://my_upstream; } location ~ .*\.(jpg|JPG|gif|GIF|png|PNG|swf|SWF|css|CSS|js|JS|inc|INC|ico|ICO) { # Server proxy cache proxy_cache my_cache; proxy_ignore_headers Cache-Control; proxy_cache_valid any 30m; # Client cache expires 30d; ... } } proxy_ignore_headers ignores Cache-Control header from a client. In this case, proxy_cache_valid is required. expires sends Cache-Control: max-age=\d+ and Expires header in order to cache on browsers. References
March 19, 2020
HTTP/2 for Go
http package in golang supports HTTP/2 protocols. It’s automatically configured. The features of HTTP/2In order to understand the benefits of HTTP/2, this document, provided by google, is helpful. It supports a lot of features including followings. binary format communications streaming messages multiplexing server push Some examples are written in this page, and I checked it so much. Streaming main.go is an example for server streaming, including a server and a client codes. The important part is calling Flush method, whose interface is http.Flusher and implemented by http.ResponseWriter. When this method is used, the buffered data on the server is sent to the client.
March 15, 2020
Introduction to GCP Cloud CDN
Target upstream servicesCloud CDN can have only GCP load balancer as the upstream services. And GCP load balancer can configure one of followings for backends. Backend services GCE instance groups GCE Network endpoint groups. These are the groups of VM instances Backend buckets See official document for the architecture using this. ConfigurationsCache can be controlled by response headers of origin servers, like cache expiration times, like Cache-Control: max-age header. The details are described in here.
November 19, 2019
Protocol Buffers for Go with Gadgets
gogo/protobuf is the library to store some extensions from golang/protobuf in this repository. There are some useful packages that golang/protobuf does not provide. For instance, Custom tags Getting StartedIn this example, protoc-gen-gogo is used but there are other binaries. To install it, you have to run: go get -u github.com/gogo/protobuf/protoc-gen-gogo Use custom tags for protobufAs of Nov. 2019, adding custom tags for generated proto messages are under discussion in golang/protobuf. This comment is the formal proposal for it. Until it’s released, gogo/protobuf can be used for such a purpose. The example is like below, which is stored here
November 19, 2019
Terraform for GCP
TroubleshootingsGKE To use kubernetes provider Got Error: Post https://[ip_address]/api/v1/namespaces/[namespace]/secrets: x509: certificate signed by unknown authority. This is related GitHub issue. google_container_cluster resource outputs cluster_ca_certificate, client_key, or client_certificate and they are base64 encoded. Secrets cannot be created Got Error: secrets is forbidden: User "system:anonymous" cannot create resource "secrets" in API group "" in the namespace "[namespace]" This is unresolved. If you run the terraform localy and you have kubernetes credentials, it may be because you do not have cluster admin roles. But if you try to create a cluster at the same time with adding secrets by terraform on CI, it seems no way yet. There is an issue on GitHub related with this error, but no good solution yet.
November 19, 2019
Terraform overview
Basic conceptsThere are some basic components for terraform. Resource: one infrastructure element, like a virtual machine. Resource type is defined in a provider. Provider: a collection of resource types. Most provider offer resource types on a platform. The example of a provider is AWS, GCP, or Kubernetes. Module: a set of resources. In order to use modules, following elements may be used. Input variables:, defined as variable. Output variables: defined as output. Local variables: defined in locals. Data Sources: Data defined and used externally from current terraform. This may be a different terraform state. State The current infrastructure configuration is stored in state by terraform. Terraform stores the state in terraform.tfstate localy by default, but using backend, state can be stored remotely like AWS S3. The state can contain sensitive data such as a database password. To avoid this, by some backends, encryption or other features are provided.
November 19, 2019
Getting Started with Google closure library
Some JavaScript library depends on Google Closure. If you need to understand the behavior of such a library, you have to know closure. The official document of closure library is here. And using Closure compiler, we can bundle codes with those libraries. There is npm version to use such a compiler. Examples ./index.html Sample to use google closure library hello.js first sample using google cloud compiler myproject/start.js, myproject/klass.js Sample using module TODO Sample to use gooogle closure library and compile Google closure compilerThe compiler can be downloaded by npm.
October 19, 2019