Overview: (Left) Major components(e.g., Text, Prompt, Compressed Pose, etc.) of Prompt2Sign dataset. Compressed Pose is reprocessed pose data that is suitable for training, we use public sign language videos to produce compressed pose data in our predefined format; (Right) Our proposed SignLLM aims to generate sign language poses for various application scenarios.

Paper

Prompt2Sign (tools)

Additional details

Methodology

Code (stay tuned)

(Left) MLSF contains parallel Enc-Dec groups (i.e., Text2Pose × number of languages), the Prompt2LangGloss adds a language attribute marker at the gloss channel (i.e., Text2Gloss2Pose → Prompt2LangGloss2Pose). (Right) The output of SignLLM can be converted into a skeletal pose video, which can then be rendered into a realistic human appearance by vid2vid models.

Comparison of different models of SignLLM with baseline. Models M and T represent MLSF and Text2LangGloss trained ASLP models, respectively.

Comparison of different models of SignLLM with previous work.

SignLLM-40M-Base results for Text to Pose on the ASL part of Prompt2Sign, with multiple data augmentation techniques. Base: Multi-Language Switching Framework with Normal MSE Loss, FP: Future Prediction, GN: Gaussian Noise, Text2LangGloss: Text2LangGloss with Normal MSE Loss, PLC: Priority Learning Channe

the comparison of the effect of different Settings on DTW values (the lower the better) at different times is determined by epoch.

### JSON file:
{
    "version": 1.3,
    "people": [
        {
            "person_id": [
                -1
            ],
            "pose_keypoints_2d": [
                666.535,
                219.883,
                ...
	        # We only get eight key points in the upper body(24 number).
            ],
            "face_keypoints_2d": [
                624.108,
                204.299,
                0.813047,
                # We don't train the face.
            ],
            "hand_left_keypoints_2d": [
                694.829,
                455.679,
                0.352355,
                ...
		# 21 key points, 63 values.
		
            ],
            "hand_right_keypoints_2d": [
                516.344,
                348.757,
                0.0184774,
                ...
		// 21 key points, 63 values.
            ],
            "pose_keypoints_3d": [],
            "face_keypoints_3d": [],
            "hand_left_keypoints_3d": [],
            "hand_right_keypoints_3d": []
        }
    ]
}
				

def getTXT(key, fnameIn, i, mode):
  output_file = f"out_data/{mode}/{key}/demo5.txt"
  if os.path.exists(output_file):
            print(f"Skipping {key} as output file already exists.")
            return
  else:
      # Getting our structure of skeletal model.
      # For customizing the structure see a definition of getSkeletalModelStructure. 
      dtype = "float32"
      randomNubersGenerator = numpy.random.RandomState(1234) 
      structure = skeletalModel.getSkeletalModelStructure()
      with h5py.File(fnameIn, "r") as hfIn:
            print("")
            print("Now is processing the "+ key)
            print("")
            inputSequence_2D = numpy.array(hfIn.get(key))

            # Decomposition of the single matrix into three matrices: x, y, w (=likelihood)
            X = inputSequence_2D
            print(X.shape)
            Xx = X[0:X.shape[0], 0:(X.shape[1]):3]
            Xy = X[0:X.shape[0], 1:(X.shape[1]):3]
            Xw = X[0:X.shape[0], 2:(X.shape[1]):3]

            # Normalization of the picture (x and y axis has the same scale)
            Xx, Xy = pose2D.normalization(Xx, Xy)
            save(f"out_data/{mode}/{key}/demo1.txt", [Xx, Xy, Xw])

            # Delete all skeletal models which have a lot of missing parts.
            Xx, Xy, Xw = pose2D.prune(Xx, Xy, Xw, (0, 1, 2, 3, 4, 5, 6, 7), 0.3, dtype)
            save(f"out_data/{mode}/{key}/demo2.txt", [Xx, Xy, Xw])

            # Preliminary filtering: weighted linear interpolation of missing points.
            Xx, Xy, Xw = pose2D.interpolation(Xx, Xy, Xw, 0.99, dtype)
            save(f"out_data/{mode}/{key}/demo3.txt", [Xx, Xy, Xw])

            # Initial 3D pose estimation
            ... = pose2Dto3D.initialization(
                ...
              )
            save(f"out_data/{mode}/{key}/demo4.txt", [Yx0, Yy0, Yz0])

              # Backpropagation-based filtering
            Yx, Yy, Yz = pose3D.backpropagationBasedFiltering(
                ...
              )
            save(f"out_data/{mode}/{key}/demo5.txt", [Yx, Yy, Yz])
            # Called after each folder is processed to release GPU resources
            print(f"Now we're working on folder {i} th, and will be finished")
            i=i+1
            tf.keras.backend.clear_session()
			

SignLLM: Sign Languages Production Large Language Models arXiv, 2024. (arXiv)

(Additional Details/ Supplementary Materials)

@misc{fang2024signllm,
      title={SignLLM: Sign Languages Production Large Language Models}, 
      author={Sen Fang and Lei Wang and Ce Zheng and Yapeng Tian and Chen Chen},
      year={2024},
      eprint={2405.10718},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}