Monday, Apr. 24th, 2023 - Sunday, Apr. 30th, 2023

2. Learning Records

2.1. Updating

Revised my English Curriculum Vitae and Chinese Resume based on the SATA principles.

2.2. Submission

Used Grammarly to correct my paper draft and created a LaTeX manuscript based on the template.

Spent about a whole afternoon writing a Cover Letter.

The manuscript was submitted to Applied Sciences on Apr. 29th, 2023.

3. Feeling

3.1. Relieved

Gave a gasp of relief after the submission. At least I took the first step.

Monday, Apr. 17th, 2023 - Sunday, Apr. 23th, 2023

2. Learning Records

2.1. Exploring

Searched for potential supervisors in CityU and PolyU.

2.2. Revision

Revised the patent document. At least four versions of the document were made.

2.3. Writing

Started to write my English Curriculum Vitae and Chinese Resume.

3. Feeling

3.1. Relieved

I gave a gasp of relief after the table which stores the information of the supervisors was filled.

Monday, April 10th, 2023 - Sunday, April 16th, 2023

2. Learning Records

2.1 Writing

Added more details about the overall architecture. Demonstrated and calculated the feature size of each stage as the network gets deeper. Besides, I also wrote the ablation part. But to be honest, writing the portion I mentioned above put me in a difficult position because it was all about bragging. I struggled with exaggerating how excellent my proposed method was. Actually, it was just slightly better than a piece of shit. Based on this fact, it was much easier to write the limitations.

2.2 Exploring

Searched for potential supervisors in HKUST and CUHK. The research areas and interests of the professors at HKUST are much more related to science, not engineering. Compared to science, I am keener on engineering.

3. Feelings

3.1 Procrastination

The students on every team are brilliant. They all come from prestigious schools. Consequently, it is almost impossible for me, a nobody, to get that advanced doctoral opportunities.

Monday April 3rd, 2023 - Saturday, April 8th, 2023

2. Learning Record

2.1 Writing

Only wrote the results of my paper drafts.

3. Feeling

3.1 Busy

I went home twice this week. It cost about half a day to go home. Going back to school from home costs the same.

Monday, March 27th, 2023 - Sunday, April 2nd, 2023

2. Learning Records

2.1 Experiments

Redid macro-expression spotting experiments on CAS(ME)^2 and SAMM Long Videos datasets. Combining different versions of macro- and micro-expression spotting experiment outcomes resulted in different performances. Luckily, There was some improvement in the overall analysis.

2.2 Building

Built the .py file that combines extraction and pre-processing together. It seemed to work well.

2.3 Writing

Wrote parts of my paper draft which demonstrates Swin Transformer, Shifted Patch Tokenization, Locality Self-Attention, and spotting.

2.4 Exploring

Searched for potential supervisors in HKU and HKUST.

3. Feeling

3.1 Irritable

It was damn humid. Everything was damn wet.

Monday, March 20th, 2023 - Friday, March 24th, 2023

2. Learning Record

2.1 Sketches Drawing

Drew illustration sketches for the patent.

2.2 Testing for MEGC 2022 Benchmarks

The performance of both Swin Transformer and SL-Swin is unstable. I had made the same experiment several times and it ended up with different outcomes.

2.3 Building

Built S-Swin and L-Swin. Did experiments using S-Swin, L-Swin, ViT, and SL-ViT.

2.4 Post-processing

I tried to apply different smoothing methods to the predicted scores but they all degraded the performance.

3. Feeling

Worried

The performance of the models on the MEGC 2022 was terrible.

Tuesday, March 13th, 2023 - Sunday, March 19th, 2023

2. Learning Records

2.1 Applying Multithreading

Applied multithreading on the overall analysis. ME analysis and MaE analysis are processed on two threads separately.

Applied multithreading on ME analysis.

2.2 Refactoring

Refactored the spotting and evaluating as a function, which made the notebook more concise.

2.3 Words Processing

Kept writing the patent paper. Added formulas, sketch images, and the reason why Swin Transformer, Shifted Patch Tokenisation and Locality Self-Attention work fine.

Nevertheless, paperwork is some kind of worth as I found a redundant softmax layer was applied to the model when I was writing the patent demonstration.

2.4 Fixing

Deleted the redundant softmax layer.

2.5 Featuring

Updated the pytorch to version 2.0 and tried torch.compile. Unfortunately, it failed. Maybe the reason is that my model is running in eager mode.

3. Feeling

Exhausted and Tired

It seemed that I was not recovered mentally and physically from the hiking last weekend.

Sunday, March 5th, 2023 - Saturday, March 11th, 2023

2. Learning Records

2.1 Refactoring

Let all hyperparameters be set at the beginning of the jupyter notebook.

Refactored the torch functions. Made the progress bar present every five epochs.

2.2 Featuring

Added a setting parameter verbose to the torch functions. When verbose = 0, the progress bar would present every split.

2.3 Debugging

I found the overall analysis was wrong as the same k was fed into the mae spotting and me spotting.

3. Feelings

3.1 Confused and Perplexed

It seems that almost all experiments are done. I have no idea.

Monday, February 27th, 2023 - Saturday, March 4th, 2023

2. Learning Records

2.1 Re-cropping

It could be concluded that I fixed the cropping problem or I added a new cropping feature. The height of the ROI returned by cv2 is slightly longer than the weight. Therefore, I added half of the height to the weight and I could get a square bounding box.

Then, the cropped square image is fed to the pre-processing. Suprisingly, the landmarks could be detected on every image, which means the detection index shift that I spent a lot of time and effort before is useless now. It was a good news or a bad news somehow.

The new cropping function is applied to both CAS(ME)^2 and SAMM Long Videos dataset.

2.2 Building

Built the training and test function for CAS_Test and SAMM_Test dataset that are used in the MEGC2022.

Built a function to store the normalize, augment and .pkl files loading function.

Built training and test .ipynb files for the MEGC2022.

2.3 Extraction and Pre-processing

Thanks to the new cropping function the extraction and pre-process were running smoothly.

2.4 Refactoring

Let hyperparameters set at the beginning of every .ipynb file. Now I could just set the hyperparameters and then click "Run All" and I could get the prediction.

2.5 Paper Writing

Wrote the part of describing the Locality Self-Attention and Swin Transformer.

3. Feelings

3.1 Confused, Perplexed and Anxious

Every macro-expression spotting experiments cost more than three days. What else could I do during the training period?

Sunday, February 19th, 2023 - Saturday, February 25th, 2023

2. Learning Records

2.1 Fixed the Crash Error

The jupyter_core and jupyter_client are alright. But I found out that the Linux server runs out of memory when the macro- and micro-expression spotting experiments are in process simultaneously.

2.2 Extracted Features

Extracted micro-expression features on the SAMM Long Videos dataset.

2.3 Wrote my Paper

Wrote the Related Work part.

2.4 Fixed the Count Mismatch Problem

Due to the problematic naming in the original cropping function, not only the images' names were wrong, but also the cropped image count mismatched the original image count. I fixed the naming problem last week and fixed the mismatch count problem this week.

In addition, I tried to extract features from the video 016_7 but the machine crashed twice because of the limited RAM. Damn! Each extraction experiment cost me more than an hour, which means I wasted about three hours that afternoon.

2.5 Added New Features

Allowed assigning specific GPU to do the training. Besides, Added adjustments to restrict TensorFlow to only allocate specific memory on the GPU.

2.6 Building

Built the test dataset cropping function, training and test function. And I combined all processes in one .ipynb file.

Moreover, I built the function for features extraction and pre-processing which could save the .pkl file after finishing extracting and pre-processing one video.

2.7 Refactoring

Added a parameter debug_preds to each .ipynb file. When debug_preds is True, the training would not be processed and the program would load the .pkl prediction files and execute the spotting and evaluation process.

2.8 Fixing the Model

It was when I was writing my paper and reading the vision transformer for small scall datasets that I found out the Shifted Patch Tokenization has a process called pos_embedding which I forgot to add in my SL models.

I added the pos_embedding on both the TensorFlow and PyTorch models.

2.9 Fixing the Cropping

I found that the dlib face detector could not detect the face landmark on every image of the test video. This weird situation let me reconsider whether my cropping function has some problem. Consequently, I changed the cropping scale.

The good news was everything worked fine and the bad news was I needed to crop all images again which was time-consuming.

3. Feelings

No special feelings.

Coding is more intriguing than paper work.

Monday, February 13th, 2023 - Saturday, February 18th, 2023

2. Learning Records

2.1 Reinstalled CUDA

All of a sudden, the CUDA on my laptop died. I spent a day debugging and reinstalling the drivers and CUDA.

2.2 Wrote the Patent Document

I spent two days writing the document. I found it an excellent way to review my work.

2.3 Building

I built the function for the ablation study of p.

2.4 Fixing

Fixed the rename error in SAMM Long Videos dataset. The sequence number of the last picture in the016_7 folder is 1000. But the original cropping function renames it 0000.jpg which makes it the first picture. Consequently, the image sequence is totally wrong.

Luckily, I found this error and fixed all the naming problem.

2.5 Started to Do Experiments on the SAMM Long Videos Dataset

I started to do the features processing on the SAMM Long Videos Dataset.

2.6 Fixed the Jupyter Error

I upgraded the jupyter packages on the Linux server. But when I start the training, close the remote desktop connection, and reconnect, I always encounter the problem of reconnecting to the kernel. And the restart function is also not available.

I am sure the new jupyter packages caused the problem. As a consequence, I downgraded the packages and then everything was back to normal.

This stupid problem wasted me more than two days.

2.7 Start to Write the Paper

Maybe it was not writing, it was replicating.

3. Feelings

Anxious

The clock is ticking and I am worried that I could not publish my paper on time.

Monday, February 6th, 2023 - Saturday, February 11th, 2023

2. Learning Records

2.1 Refactor for PyTorch

Refactor the code using PyTorch. But it outputs terrible results. I spent two days debugging trying different epochs and batch_size. Finally, I found the validation dataset takes X_train and X_test as inputs, which is the reason for the bad outcomes.

2.2 Build Models

Build swin transformer and sl swin transformer using PyTorch.

2.3 Fix the Accuracy Calculation

The calculation is not suitable for the output of one dimension. Just fixed it.

2.4 Record the Results

Finish one macro expression spotting experiment, which cost 44 hours. At least the result is not bad. I hope I could get better outcomes through some fine-tuning.

3. Feeling

3.1 Perplexed

44 hours for one experiment. What can I do?

Tuesday, January 31st, 2023 - Saturday, February 4th, 2023

2. Learning Record

2.1 Refactor for Macro Expression Spotting

Save the resampled features of every subject separately to lift the restrictions of the tf.data buffer size.

Save different versions of .ipynb files and rename them into generator, backup, and dev.

2.2 Refactor for LOSO

Refactor the code by using sklearn.model_selection.LeaveOneGroupOut to two for loops.

Theoretically, it should give almost the same outputs. But the outputs from the for loops are slightly worse than those from the LeaveOneGroupOut. And the training time is about two times of the file using LeaveOneGroupOut. Damn!

2.3 Refactor the Spotting and Evaluation

Put the prediction parameter preds out of the training.py file and do the spotting and evaluation when all training loops are finished. By doing this, I can debug the spotting and evaluation without repeating the training step.

3. Feelings

3.1 Perplexed

The training for macro expression spotting costs more than a day. I don't know what to do.

Monday, January 2nd, 2023 - January 7th, 2023

2. Learning Records

2.1 Redid the Experiments

Okay, I found the reason why the model ran like shit. The shuffle of the validation dataset made terrible results. And different batch_size or normalizations resulted in different outcomes.

2.2 Refactoring

The error said I encountered the 2 G limit of the tensorflow dataset buffer size. I tried to use the from_generator function. Unfortunately, it worked worse. I can't even finish the micro-expression training.

Furthermore, the learning rate seemed to have a huge impact on the outcomes. So whether the model is shit or the learning rate doesn't fit. I had no idea.

3. Feelings

3.1 Relieved

Finally, I finished the NTCE test on Saturday, January 7th, 2023. Then I could focus more on the experiment.

Monday, December 26th, 2022 - Saturday, December 31st, 2022

2. Learning Records

2.1 Redid the Cropping

I tried Haar Cascade face detector in cv2, deep learning based face detector in cv2, HoG face detector in dlib, deep learning based dlib face detection. The dnn face detector worked the best. The others have prominent position shifting on different images.

2.2 Set Up the Remote Repeater

I installed RustDesk on the repeater computer and left it in school. So I could use RustDesk to connect to the repeater and then connect to the server even when I am at home.

2.3 Repeated the Experiments

The model ran as shit, even the origin model SOFTNet.

3. Feelings

3.1 Worried and Happy

The new year was coming!

Sunday, December 18th, 2022 - Saturday, December 24th, 2022

2. Learning Record

2.1 Learned Linux

I learned the terminal commands and learned to use WinSCP, and Putty. Besides, I learned to use the remote SSH interpreter. I could debug the code on my laptop. It runs smoothly as it doesn't need to take the delay from the remoter desktop. But it cost several seconds for saving. Weird.

2.2 Refactoring

Replace glob with pathlib.

Debugged the code on Linux. The biggest batch_size could be set to 128, which saved a huge amount of time during training. I only did the training on the Windows server once. Theoretically, it needed 36 h for one time of training. But as other users started to use the server, the 3D calculation part of the GUP was occupied 100%. Without a doubt, the server stuck. So actually, I didn't finish even one time of training on the Windows server.

2.3 Build A New Model

Built the sl-swin-transformer.

3. Feeling

3.1 Tired or Exhausted

It seemed that I got COVID-19 on December 19th, 2022. I needed to prepare for the NTCE exam as well as do experiments.

Monday, December 12th, 2022 - Saturday, December 17th, 2022

2. Learning Record

2.1 Added CBAM

I added cbam [1] on the SOFTNet, but it ran as shit. I tried some fine-tuning, but I still got terrible results.

2.2 Refactored tf.dataset

I tried to use the generator to generate data for the vision transformer model, but the yield would yield a tensor in the shape of [none, none, none, none] which causes errors.

2.3 Request for the Test Dataset

I sent the request e-mail and got the reply with the link to the dataset within one morning, the morning of December 15th, 2022.

The efficiency shocked me a lot!

2.4 Deploy on New Server

The training made the Windows server really slow even stuck. Consequently, I got a new account for the Linux server. It has two GTX 2080 Ti GPUs, cool!

2.5 Paper Reading

I read the paper [2]. It uses traditional methods but got the first prize in the competition. Unbelievable!

3. Feelings

3.1 Busy

The code runs as shit and I needed to learn how to use Linux.

References

[1]S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “CBAM: Convolutional Block Attention Module,” in Computer Vision – ECCV 2018, Cham, 2018, pp. 3–19.

[2]J. Yu, Z. Cai, Z. Liu, G. Xie, and P. He, “Facial Expression Spotting Based on Optical Flow Features,” in Proceedings of the 30th ACM International Conference on Multimedia, New York, NY, USA, 2022, pp. 7205–7209. doi: 10.1145/3503161.3551608.

1. Duration

Monday, December 5th, 2022 - Saturday, December 10th, 2022

2. Learning Record

2.1 Learned Capsule Network

Read the paper [1] and learned the code. Refactor the code to fit the input of different datasets. Also, I turned the eager mode off.

2.2 Learned Efficient Capsule Network

Read the paper [2] and learned the code. The network attained fabulous results on the fashion MNIST dataset in only five epochs. But the loss quickly turns nan in other datasets. Besides, the computation power of my laptop is enough to do the training.

2.3 Learned the CBAM

Read the paper [3] and learned the code. Compared to the vision transformer, this code is a lot easier.

3. Feeling

3.1 Anxious

I had some new ideas about micro-expression spotting, but it seemed I don't have much time.

4. References

[1]S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic Routing Between Capsules.” arXiv, 2017. doi: 10.48550/ARXIV.1710.09829.

[2]V. Mazzia, F. Salvetti, and M. Chiaberge, “Efficient-CapsNet: capsule network with self-attention routing,” Scientific Reports, vol. 11, no. 1, p. 14634, Jul. 2021, doi: 10.1038/s41598-021-93977-0.

[3]S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “CBAM: Convolutional Block Attention Module,” in Computer Vision – ECCV 2018, Cham, 2018, pp. 3–19.

1. Duration

Monday, November 28th, 2022 - Saturday, December 3th, 2022

2. Learning Record

2.1 Refactored the Code Using PyTorch

Refactored the vision transformer for small-size datasets code using pytorch. The code is more comprehensible but the dataset and dataloader part is more difficult. The model part is easier but the training part which shows the training process takes some additional work. I needed to use the tqdm to manually build a process bar and make the training process run.

Similar to the tf_data_constructor, I also build a torch_data_constructor python file to construct the dataset and dataloader. Consequently, only the base_dir needed to be modified.

2.2 Learn the Capsule Network

Read the paper [1] and watch videos about it. Also, I start to learn the code.

2.3 Refactored the TensorFlow Code

Turned the code back to the default graph mode. It seemed to be faster and can handle a bigger batch_size.

3. Feeling

3.1 Excited

After a lockdown for approximately fifteen days, I finally went back to the lab on November 30th.

3.2 Unfamiliar

Everything in the laboratory seemed so unfamiliar to me.

Monday, November 21st, 2022 - Saturday, November 26th, 2022

2. Learning Record

2.1 Fine-tuned Models

When images have only two classes which means the labels are "0" and "1", the `label_mode` of `tf.keras.utils.image_dataset_from_directory` should be `binary` and the loss_fn should be `BinaryCrossentropy` instead of `SparseCategoricalCrossentropy`. Otherwise, the accuracy will jitter around 50%, which means the model learns nothing.

2.2 Learned Swin Transformer

I watched the paper [1] and read the code.

It took me a few days to understand the `Window-based Self-Attention & Shifted Window-based Self-Attention` and the `Swin Transformer Block`.

2.3 Refactored the Code

I built a py file to store all functions for loading the datasets. Consequently, the code could work only by changing the basee_dir of the dataset.

3. Feeling

3.1 Glad

I was glad that the models ran well.

3.2 Perplexed

The models seemed to generate relatively good results on some small datasets. But they didn't work that well on the micro-expression dataset.

Besides ViT, SL-ViT, Swin Transformer, I also found that there are lots of other types of transformer models. It seems impossible to learn all of them.