Hello Tachibana san, Great work and congratulations on the work on AnimeGANv2. I had good success in converting the models and running the models on Android. However the latency is still an issue, it takes about 500 ms to run a 128x128 patch of image using Tensorflow Android(I tried tflite but it increases the inference time strangely.) I want to modify the network architecture and optimize its performance further to make it a real-time application (under 100 ms) So to cut a long story short, are you planning to release the training code in near future? :)

Thank you.

Hello, thank you for posting this great work!

I have retrained the model with a customized dataset, the results look great but the loss curves seem strange to me.

The adversary loss seems ok, I set the weights for D and G to 200 and 300, respectively, and the losses are approaching the equilibrium.

However, the G_vgg loss, which consists of c_loss, s_loss, color_loss, tv_loss, reaches the bottom at around epoch 30 and then starts increasing. I looked at each individual loss among G_vgg_loss, only the s_loss is decreasing over time, and all others starts increasing after epoch 30.

Interestingly, the validation samples from epoch 100 is apparently better than the ones from epoch 30. Does anyone experience the same?

Hi, I Try To run this Project But I have little problem and when I try to start train phase this code is just only run on cpu but I install cudatoolkit and tensorflow-gpu. can u help me ?

2020-12-06 18:00:45.278352: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA     [693/1792]
2020-12-06 18:00:45.302721: E tensorflow/stream_executor/cuda/cuda_driver.cc:397] failed call to cuInit: CUDA_ERROR_NO_DEVICE                                                              
2020-12-06 18:00:45.302766: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: host-name
2020-12-06 18:00:45.302775: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: host-name
2020-12-06 18:00:45.302816: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 450.66.0
2020-12-06 18:00:45.302853: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 450.66.0                                                                 2020-12-06 18:00:45.302863: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:300] kernel version seems to match DSO: 450.66.0

npy file loaded -------  vgg19_weight/vgg19.npy
##### Information #####
# gan type :  lsgan
# light :  False
# dataset :  Hayao
# max dataset number :  6656
# batch_size :  12
# epoch :  101
# init_epoch :  10
# training image size [H, W] :  [256, 256]
# g_adv_weight,d_adv_weight,con_weight,sty_weight,color_weight,tv_weight :  300.0 300.0 1.5 2.5 10.0 1.0
# init_lr,g_lr,d_lr :  0.0002 2e-05 4e-05
# training_rate G -- D: 1 : 1
build model finished: 0.138872s
build model finished: 0.130571s
build model finished: 0.120662s
build model finished: 0.127711s
build model finished: 0.123440s
G:
---------
Variables: name (type shape) [size]
---------
generator/G_MODEL/A/Conv/weights:0 (float32_ref 7x7x3x32) [4704, bytes: 18816]
generator/G_MODEL/A/LayerNorm/beta:0 (float32_ref 32) [32, bytes: 128]
generator/G_MODEL/A/LayerNorm/gamma:0 (float32_ref 32) [32, bytes: 128]
generator/G_MODEL/A/Conv_1/weights:0 (float32_ref 3x3x32x64) [18432, bytes: 73728]
generator/G_MODEL/A/LayerNorm_1/beta:0 (float32_ref 64) [64, bytes: 256]
generator/G_MODEL/A/LayerNorm_1/gamma:0 (float32_ref 64) [64, bytes: 256]
generator/G_MODEL/A/Conv_2/weights:0 (float32_ref 3x3x64x64) [36864, bytes: 147456]
generator/G_MODEL/A/LayerNorm_2/beta:0 (float32_ref 64) [64, bytes: 256]
generator/G_MODEL/A/LayerNorm_2/gamma:0 (float32_ref 64) [64, bytes: 256]
generator/G_MODEL/B/Conv/weights:0 (float32_ref 3x3x64x128) [73728, bytes: 294912]
generator/G_MODEL/B/LayerNorm/beta:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/B/LayerNorm/gamma:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/B/Conv_1/weights:0 (float32_ref 3x3x128x128) [147456, bytes: 589824]
generator/G_MODEL/B/LayerNorm_1/beta:0 (float32_ref 128) [128, bytes: 512]                                                                                                        [649/1792]generator/G_MODEL/B/LayerNorm_1/gamma:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/C/Conv/weights:0 (float32_ref 3x3x128x128) [147456, bytes: 589824]
generator/G_MODEL/C/LayerNorm/beta:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/C/LayerNorm/gamma:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/C/r1/Conv/weights:0 (float32_ref 1x1x128x256) [32768, bytes: 131072]
generator/G_MODEL/C/r1/LayerNorm/beta:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r1/LayerNorm/gamma:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r1/r1/w:0 (float32_ref 3x3x256x1) [2304, bytes: 9216]
generator/G_MODEL/C/r1/r1/bias:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r1/1/beta:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r1/1/gamma:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r1/Conv_1/weights:0 (float32_ref 1x1x256x256) [65536, bytes: 262144]
generator/G_MODEL/C/r1/2/beta:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r1/2/gamma:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r2/Conv/weights:0 (float32_ref 1x1x256x512) [131072, bytes: 524288]
generator/G_MODEL/C/r2/LayerNorm/beta:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r2/LayerNorm/gamma:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r2/r2/w:0 (float32_ref 3x3x512x1) [4608, bytes: 18432]
generator/G_MODEL/C/r2/r2/bias:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r2/1/beta:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r2/1/gamma:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r2/Conv_1/weights:0 (float32_ref 1x1x512x256) [131072, bytes: 524288]
generator/G_MODEL/C/r2/2/beta:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r2/2/gamma:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r3/Conv/weights:0 (float32_ref 1x1x256x512) [131072, bytes: 524288]
generator/G_MODEL/C/r3/LayerNorm/beta:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r3/LayerNorm/gamma:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r3/r3/w:0 (float32_ref 3x3x512x1) [4608, bytes: 18432]
generator/G_MODEL/C/r3/r3/bias:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r3/1/beta:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r3/1/gamma:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r3/Conv_1/weights:0 (float32_ref 1x1x512x256) [131072, bytes: 524288]
generator/G_MODEL/C/r3/2/beta:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r3/2/gamma:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r4/Conv/weights:0 (float32_ref 1x1x256x512) [131072, bytes: 524288]
generator/G_MODEL/C/r4/LayerNorm/beta:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r4/LayerNorm/gamma:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r4/r4/w:0 (float32_ref 3x3x512x1) [4608, bytes: 18432]
generator/G_MODEL/C/r4/r4/bias:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r4/1/beta:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r4/1/gamma:0 (float32_ref 512) [512, bytes: 2048]
generator/G_MODEL/C/r4/Conv_1/weights:0 (float32_ref 1x1x512x256) [131072, bytes: 524288]
generator/G_MODEL/C/r4/2/beta:0 (float32_ref 256) [256, bytes: 1024]
generator/G_MODEL/C/r4/2/gamma:0 (float32_ref 256) [256, bytes: 1024]                                                                                                             [605/1792]generator/G_MODEL/C/Conv_1/weights:0 (float32_ref 3x3x256x128) [294912, bytes: 1179648]
generator/G_MODEL/C/LayerNorm_1/beta:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/C/LayerNorm_1/gamma:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/D/Conv/weights:0 (float32_ref 3x3x128x128) [147456, bytes: 589824]
generator/G_MODEL/D/LayerNorm/beta:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/D/LayerNorm/gamma:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/D/Conv_1/weights:0 (float32_ref 3x3x128x128) [147456, bytes: 589824]
generator/G_MODEL/D/LayerNorm_1/beta:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/D/LayerNorm_1/gamma:0 (float32_ref 128) [128, bytes: 512]
generator/G_MODEL/E/Conv/weights:0 (float32_ref 3x3x128x64) [73728, bytes: 294912]
generator/G_MODEL/E/LayerNorm/beta:0 (float32_ref 64) [64, bytes: 256]
generator/G_MODEL/E/LayerNorm/gamma:0 (float32_ref 64) [64, bytes: 256]
generator/G_MODEL/E/Conv_1/weights:0 (float32_ref 3x3x64x64) [36864, bytes: 147456]
generator/G_MODEL/E/LayerNorm_1/beta:0 (float32_ref 64) [64, bytes: 256]
generator/G_MODEL/E/LayerNorm_1/gamma:0 (float32_ref 64) [64, bytes: 256]
generator/G_MODEL/E/Conv_2/weights:0 (float32_ref 7x7x64x32) [100352, bytes: 401408]
generator/G_MODEL/E/LayerNorm_2/beta:0 (float32_ref 32) [32, bytes: 128]
generator/G_MODEL/E/LayerNorm_2/gamma:0 (float32_ref 32) [32, bytes: 128]
generator/G_MODEL/out_layer/Conv/weights:0 (float32_ref 1x1x32x3) [96, bytes: 384]
Total size of variables: 2143552
Total bytes of variables: 8574208
 [*] Reading checkpoints...
 [*] Failed to find a checkpoint
 [!] Load failed...
Epoch:   0 Step:     0 /   554  time: 80.342747 s init_v_loss: 592.22143555  mean_v_loss: 592.22143555

Hey @TachibanaYoshino 👋, This PR aims to add basic Weights and Biases Metric Logging by appending to the existing codebase with minimal changes.

The changes can be summarized as follows :-

Add 3 extra arguments namely --use_wandb, --wandb_project and --wandb_entity which can be used to specify whether to use wandb, the name of the project to be used ("AnimeGANv2" by default) and name of the entity to be used.

Thank you for your awesome project. I think if i can using mulitple GPUs to traning, it's making things be more efficient. Hope to get some advice from you. Thanks.

Hello Author! your folder naming has a typo though It's kinda annoying to rename it again and again cuz I'm using google colab cuz I tried to use shinkai

hello! when I try to train the model, I get the following error when the code tries to save the checkpoint:

Traceback (most recent call last):
  File "main.py", line 115, in <module>
    main()
  File "main.py", line 107, in main
    gan.train()
  File "/content/drive/.shortcut-targets-by-id/1X8hfrOWE2KxmaJG4LFKH9ydVQ4BA7oyZ/cs7643-final-project/AnimeGANv2.py", line 302, in train
    self.save(self.checkpoint_dir, epoch)
  File "/content/drive/.shortcut-targets-by-id/1X8hfrOWE2KxmaJG4LFKH9ydVQ4BA7oyZ/cs7643-final-project/AnimeGANv2.py", line 341, in save
    self.saver.save(self.sess, os.path.join(checkpoint_dir, self.model_name + '.model'), global_step=step)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py", line 1186, in save
    save_relative_paths=self._save_relative_paths)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/checkpoint_management.py", line 231, in update_checkpoint_state_internal
    last_preserved_timestamp=last_preserved_timestamp)
  File "/tensorflow-1.15.2/python3.7/tensorflow_core/python/training/checkpoint_management.py", line 110, in generate_checkpoint_state_proto
    model_checkpoint_path = os.path.relpath(model_checkpoint_path, save_dir)
  File "/usr/lib/python3.7/posixpath.py", line 475, in relpath
    start_list = [x for x in abspath(start).split(sep) if x]
  File "/usr/lib/python3.7/posixpath.py", line 383, in abspath
    cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

I mounted my google drive into colab and am using colab to train the model. When I check my checkpoint folder, I have two files there but it appears that I am missing the checkpoint binary file and the .meta file. Any idea why this could be happening?

def rgb2yuv(rgb):
    """
    Convert RGB image into YUV https://en.wikipedia.org/wiki/YUV
    """
    rgb = (rgb + 1.0)/2.0
    return tf.image.rgb_to_yuv(rgb)

tf.image.rgb_to_yuv(rgb) do the op: rgb_to_yuv so,I can't understand what this line of code means： "rgb = (rgb + 1.0)/2.0"

Hi, thanks for sharing your work. What in your opinion was the key to achieving better performance compared to your earlier model (v1) and/or other models?

I've roughly seen the code of this repo but I can't figure it out.

I want to use 512 x 512 or higher resolution images, my plan is as follows:

1. ffmpeg extracts 1080 * 1080 pictures, then sacle to 512 x 512
2. python edge_smooth.py --dataset xxxx --img_size 512
3. python train.py --dataset xxxx --epoch 101 --init_epoch 10 But I see that the pictures in train_photo under the dataset will also be used for training, so do the pictures in train_photo need to be updated to 512 x 512?

Hi, Could you share how you get these 3 improvements that you mentioned in the readme?

1. Solve the problem of high-frequency artifacts in the generated image.

2. It is easy to train and directly achieve the effects in the paper.

3. Further reduce the number of parameters of the generator network. (generator size: 8.17 Mb), The lite version has a smaller generator model.

Is the training method the same as the training landscape photos? The difference is the data set. As long as the human face data is aligned, plus the animation face alignment, is that right?