Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

I removed one subpixel convolution to upscale the picture twice, not quadruple. Then, the following error message appeared : ValueError: Dimension 2 in both shapes must be equal, but are 256 and 64. Shapes are [1,1,256,3] and [1,1,64,3]. for 'Assign_171' (op: 'Assign') with input shapes: [1,1,256,3], [1,1,64,3]. Something is wrong. Did you know how to resolve this error? please help.

python train.py

2019-10-08 20:55:32.978162: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll 2019-10-08 20:55:35.246078: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll 2019-10-08 20:55:35.333633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Quadro P1000 major: 6 minor: 1 memoryClockRate(GHz): 1.5185 pciBusID: 0000:01:00.0 2019-10-08 20:55:35.341328: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check. 2019-10-08 20:55:35.348135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2019-10-08 20:55:35.351468: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2019-10-08 20:55:35.359874: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Quadro P1000 major: 6 minor: 1 memoryClockRate(GHz): 1.5185 pciBusID: 0000:01:00.0 2019-10-08 20:55:35.366853: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check. 2019-10-08 20:55:35.372725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2019-10-08 20:55:36.070183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-08 20:55:36.075760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2019-10-08 20:55:36.079969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2019-10-08 20:55:36.084090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3005 MB memory) -> physical GPU (device: 0, name: Quadro P1000, pci bus id: 0000:01:00.0, compute capability: 6.1) 2019-10-08 20:55:36.279220: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll 2019-10-08 20:55:37.369250: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows Relying on driver to perform ptx compilation. This message will be only logged once. Traceback (most recent call last): File "train.py", line 202, in train() File "train.py", line 74, in train G = get_G((batch_size, 96, 96, 3)) File "D:\Users<Username>\Downloads\srgan-master\model.py", line 27, in get_G n = BatchNorm(gamma_init=g_init)(n) NameError: name 'BatchNorm' is not defined

packages which I noticed installing/ needed to install

tensorboard 2.0.0 tensorflow-estimator 2.0.0 tensorflow-gpu 2.0.0 tensorlayer 2.1.0

Pillow 6.2.0 google-pasta 0.1.7 Lasagne 0.1 Markdown 3.1.1

pip 19.2.3 Python 3.7.4

Os: win10 CUDA computing toolkit 10.1 and 10.0 GPU Nvidia qadro p1000 CPU: Intel core I7 8750H

Traceback (most recent call last):

File "", line 1, in runfile('/home/dongwen/Desktop/SRGAN/train.py', wdir='/home/dongwen/Desktop/SRGAN')

File "/home/dongwen/anaconda3/envs/tf2/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 827, in runfile execfile(filename, namespace)

File "/home/dongwen/anaconda3/envs/tf2/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 110, in execfile exec(compile(f.read(), filename, 'exec'), namespace)

File "/home/dongwen/Desktop/SRGAN/train.py", line 292, in train()

File "/home/dongwen/Desktop/SRGAN/train.py", line 148, in train logits_fake = D(fake_patchs)

File "/home/dongwen/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorlayer/models/core.py", line 296, in call return self.forward(inputs, **kwargs)

File "/home/dongwen/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorlayer/models/core.py", line 339, in forward memory[node.name] = node(node_input)

File "/home/dongwen/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorlayer/layers/core.py", line 431, in call outputs = self.layer.forward(inputs, **kwargs)

File "/home/dongwen/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorlayer/layers/dense/base_dense.py", line 106, in forward z = tf.matmul(inputs, self.W)

File "/home/dongwen/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 2580, in matmul a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)

File "/home/dongwen/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5753, in mat_mul _six.raise_from(_core._status_to_exception(e.code, message), None)

File "", line 3, in raise_from

InvalidArgumentError: Matrix size-incompatible: In[0]: [4,4096], In[1]: [256,1] [Op:MatMul] name: MatMul/

Traceback (most recent call last): File "train.py", line 202, in train() File "train.py", line 74, in train G = get_G((batch_size, 96, 96, 3)) File "/xx/SR/srgan-tf/model.py", line 27, in get_G n = BatchNorm(gamma_init=g_init)(n) NameError: name 'BatchNorm' is not defined

why ?

When I run python train.py --mode=evaluate command I get the following error:

Traceback (most recent call last): File "train.py", line 204, in evaluate() File "train.py", line 172, in evaluate G.load_weights(os.path.join(checkpoint_dir, 'g.h5')) File "/home/himanshu/BTP_AB/lib/python3.7/site-packages/tensorlayer/models/core.py", line 944, in load_weights raise FileNotFoundError("file {} doesn't exist.".format(filepath)) FileNotFoundError: file models/g.h5 doesn't exist.

It is may be because The Process killed after running through initializing learning (line 89 to 105 in train.py) and before adversarial learning (line 106 to 132 in tran.py).

NameError: name 'time' is not defined same as #76, #91 but I can't find a solution... Where should I add exactly ' import time' in model.py? I tried like 10 times to add import time in model.py, but it didn't work..

I am running it on google colab and I have been assigned to TESLA K80 GPU

Even 12GB ram is not suffice. I wonder if someone else is facing same problem!

Consequently the popped out error is: 2019-07-25 11:02:16.370912: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 28311552 exceeds 10% of system memory. Traceback (most recent call last): File "train.py", line 357, in train() File "train.py", line 100, in train grad = tape.gradient(mse_loss, G.weights) AttributeError: 'Model' object has no attribute 'weights'

note:

Pre-trained model vgg19.npy is in "srgan/models" directory.

I found this model is hard to train with my dataset on my machine. Is there any pretrained model? Maybe a pretrained model can save my hard-working graphics card.

Hello, author. I run your code without modification, but the resulted image has serious checkboard / mosaic shape. Could you tell me why? Have you met this problem before? How should I do to solve it? Thank you very much.

"this is original image"

"this is resulted image"

[!] Load checkpoint/g_srgan.npz failed! [!] Load checkpoint/g_srgan_init.npz failed! [!] Load checkpoint/d_srgan.npz failed! Please download vgg19.npz from : https://github.com/machrisaa/tensorflow-vgg

My training code:

    # initialize learning (G)
    n_step_epoch = round(n_epoch_init // batch_size)
    for step, (lr_patchs, hr_patchs) in enumerate(train_ds):
        step_time = time.time()
        with tf.GradientTape() as tape:
            fake_hr_patchs = G(lr_patchs)
            mse_loss = tl.cost.mean_squared_error(fake_hr_patchs, hr_patchs, is_mean=True)
        grad = tape.gradient(mse_loss, G.trainable_weights)
        g_optimizer_init.apply_gradients(zip(grad, G.trainable_weights))
        step += 1
        epoch = step//n_step_epoch
        print("Epoch: [{}/{}] step: [{}/{}] time: {}s, mse: {} ".format(
            epoch, n_epoch_init, step, n_step_epoch, time.time() - step_time, mse_loss))
        if (epoch != 0) and (step % n_step_epoch == 0):
            tl.vis.save_images(fake_hr_patchs.numpy(), [ni, ni], save_dir_gan + '/train_g_init_{}.png'.format(epoch))
        if (epoch >= n_epoch_init):
            break

    # adversarial learning (G, D)
    n_step_epoch = round(n_epoch // batch_size)
    for step, (lr_patchs, hr_patchs) in enumerate(train_ds):
        with tf.GradientTape(persistent=True) as tape:
            fake_patchs = G(lr_patchs)
            logits_fake = D(fake_patchs)
            logits_real = D(hr_patchs)
            feature_fake = VGG((fake_patchs+1)/2.)
            feature_real = VGG((hr_patchs+1)/2.)
            d_loss1 = tl.cost.sigmoid_cross_entropy(logits_real, tf.ones_like(logits_real))
            d_loss2 = tl.cost.sigmoid_cross_entropy(logits_fake, tf.zeros_like(logits_fake))
            d_loss = d_loss1 + d_loss2
            g_gan_loss = 1e-3 * tl.cost.sigmoid_cross_entropy(logits_fake, tf.ones_like(logits_fake))
            mse_loss = tl.cost.mean_squared_error(fake_patchs, hr_patchs, is_mean=True)
            vgg_loss = 2e-6 * tl.cost.mean_squared_error(feature_fake, feature_real, is_mean=True)
            g_loss = mse_loss + vgg_loss + g_gan_loss
        grad = tape.gradient(g_loss, G.trainable_weights)
        g_optimizer.apply_gradients(zip(grad, G.trainable_weights))
        grad = tape.gradient(d_loss, D.weights)
        d_optimizer.apply_gradients(zip(grad, D.trainable_weights))
        step += 1
        epoch = step//n_step_epoch
        print("Epoch: [{}/{}] step: [{}/{}] time: {}s, g_loss(mse:{}, vgg:{}, adv:{}) d_loss: {}".format(
            epoch, n_epoch_init, step, n_step_epoch, time.time() - step_time, mse_loss, vgg_loss, g_gan_loss, d_loss))

        # update learning rate
        if epoch != 0 and (epoch % decay_every == 0):
            new_lr_decay = lr_decay**(epoch // decay_every)
            lr_v.assign(lr_init * new_lr_decay)
            log = " ** new learning rate: %f (for GAN)" % (lr_init * new_lr_decay)
            print(log)

        if (epoch != 0) and (step % n_step_epoch == 0):
            tl.vis.save_images(fake_hr_patchs.numpy(), [ni, ni], save_dir_gan + '/train_g_{}.png'.format(epoch))
            G.save_weights(checkpoint_dir + '/g_{}.h5'.format(tl.global_flag['mode']))
            D.save_weights(checkpoint_dir + '/d_{}.h5'.format(tl.global_flag['mode']))
        if (epoch >= n_epoch):
            break

My error:

 File "train.py", line 370, in <module>

  File "train.py", line 125, in train
    with tf.GradientTape(persistent=True) as tape:
  File "F:\Python\Python37\lib\site-packages\tensorlayer\models\core.py", line 295, in __call__
    return self.forward(inputs, **kwargs)
  File "F:\Python\Python37\lib\site-packages\tensorlayer\models\core.py", line 338, in forward
    memory[node.name] = node(node_input)
  File "F:\Python\Python37\lib\site-packages\tensorlayer\layers\core.py", line 433, in __call__
    outputs = self.layer.forward(inputs, **kwargs)
  File "F:\Python\Python37\lib\site-packages\tensorlayer\layers\dense\base_dense.py", line 106, in forward
    z = tf.matmul(inputs, self.W)
  File "F:\Python\Python37\lib\site-packages\tensorflow\python\util\dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "F:\Python\Python37\lib\site-packages\tensorflow\python\ops\math_ops.py", line 2647, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "F:\Python\Python37\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 6285, in mat_mul
    _six.raise_from(_core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Matrix size-incompatible: In[0]: [1,18432], In[1]: [512,1] [Op:MatMul] name: MatMul/

My loading of images:

    def generator_train():
        i = 0
        while i < len(train_hr_imgs):
            yield train_hr_imgs[i], train_lr_imgs[i]
            i+=1
    def _map_fn_train(imgh, imgl):
        hr_patch = imgh
        lr_patch = imgl
        
        hr_patch = hr_patch / (255. / 2.)
        hr_patch = hr_patch - 1.
        
        lr_patch = lr_patch / (255. / 2.)
        lr_patch = lr_patch - 1.
        
        return lr_patch, hr_patch
    train_ds = tf.data.Dataset.from_generator(generator_train, output_types=(tf.float32, tf.float32))
    train_ds = train_ds.map(_map_fn_train, num_parallel_calls=multiprocessing.cpu_count())

I prescale the input images to 384 (HR) and 96 (LR)

Any idea how to fix this?

in model = eval , it shows RuntimeError: Weights named 'conv2d_1/filters:0' not found in network. Hint: set argument skip=Ture if you want to skip redundant or mismatch weights.

i download g.npz and d.npz and pretrained vgg19 in your readme

Total perceptual loss in SRGAN paper is weighted sum of content loss and adversarial loss.

Total loss = Content loss + (10^(-3)) Adversarial loss Please tell why 10^(-3) is used? what is its impact on performance if some other value is used ?? or does it affects number of iterations for training of network ?

Using TensorFlow backend. 2022-04-26 18:53:56.612121: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-04-26 18:53:56.616458: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance. Traceback (most recent call last): File "/home/wajoud/srgan/train.py", line 105, in G = SRGAN_g() File "/home/wajoud/srgan/srgan.py", line 34, in init self.residual_block = self.make_layer() File "/home/wajoud/srgan/srgan.py", line 46, in make_layer return SequentialLayer(layer_list) File "/home/wajoud/anaconda3/envs/srgan/lib/python3.9/site-packages/tensorlayerx/nn/layers/deprecated.py", line 451, in SequentialLayer raise NonExistingLayerError("SequentialLayer(layer) --> Sequential(layer)(in)" + log) tensorlayerx.nn.layers.deprecated.NonExistingLayerError: SequentialLayer(layer) --> Sequential(layer)(in) Hint: 1) downgrade TL from version TensorLayerX to TensorLayer2.x. 2) check the documentation of TF version 2.x and TL version X

facing this issue can someone help me out !! thank you

In the paper the authors have tested on Set5, Set14 and BSD dataset, was anyone able to replicate the same results ? @Laicheng0830 can you share your benchmarking results ---> Image results/ PSNR/ SSIM metric Also is it possible to share your validation loss plots ?

One thing that boils my blood is bad documentation and this project dosent lack it a little bit No really how a new user (in my case a 3d designer) should know What is bicubic LG valid and train images or what the hell is evaluating and what should I do with vgg 19 ? It really pathetic that some one made an ai but can't document it And reading the issues half of them could be easily solved with decent documentation (not good documentation)